QU-SSM-MoE

-brightgreen)

QU-SSM-130M-MoE is a continuous quasi-unitary state space sequence architecture co-designed with sparse SwiGLU Mixture-of-Experts (MoE). Designed and invented by Prannessh K.V.A. (Sole Architect & Inventor), QU-SSM eliminates monotonic exponential memory dissipation present in classical real-valued SSMs while delivering strictly constant O(1) step memory (0.19 MB) and 3.32x generation speedup over modern baseline Transformers.

🧬 Base Model Lineage & Technical Notes

• Core Architecture: Novel Continuous Quasi-Unitary Lie-Algebra Recurrent State Space Operator (SO(2) phase rotations) with 8 SwiGLU Mixture-of-Experts and Top-2 gating.
• Tokenizer Lineage: Standard GPT-2 Byte-Pair Encoding (BPE) vocabulary (50,257 tokens).
• Pre-training & Calibration: Initialized and pre-trained on roneneldan/TinyStories (~20M+ tokens) to empirically validate non-dissipative recurrent memory and narrative coherence.
• Parameter Footprint: 134.89M Total Parameters, 78.27M Active Parameters per token.
• State Space Dynamics: Strictly preserves vector norms (||R(theta)||_2 = 1.000) through dual-trigonometric rotation, eliminating contractive gradient decay across long sequence horizons.
• Official Research Contact: LinkedIn — Prannesh K. V. A.

🔍 What is QU-SSM (QU-SSM-130M-MoE)?

QU-SSM is a state-of-the-art linear-time sequence engine that formulates state transitions as continuous Lie-group rotations over SO(2) ≅ U(1). Unlike classical real-decay state space models that suffer from exponential forgetting, QU-SSM preserves state norm (‖R(θ)‖₂ ≡ 1.00000) while dynamically decoupling memory damping (γ_t).

Key Advantages of QU-SSM-130M-MoE:

• ⚡ 3.32x Faster Inference: Generates at 5.55 tok/s (180.16 ms/tok step latency) compared to 1.67 tok/s for standard dense Transformer baselines.
• 💾 Constant O(1) Memory: Requires strictly 0.19 MB of state RAM regardless of context length (1,894x smaller than a 360 MB KV-cache at L = 8,192).
• 🧠 42% Lower Active Compute: Evaluates only 78.27M active parameters per token across 8 SwiGLU experts with Top-2 routing.
• 🌊 Lowest Spectral Phase Distortion: Demonstrates 1.9648 rad phase error on continuous sensor and financial telemetry streams.

🏆 SOTA Benchmark Leaderboard (~135M Scale)

🔬 Mathematical Formulation of QU-SSM

The core recurrence equation evolves according to:

h_t = γ_t · R(θ_t) · h_{t-1} + u_t

Where:

• R(θ_t) ∈ SO(2) is a 2x2 rotation matrix with strict norm preservation: ‖R(θ_t)‖₂ ≡ 1.00000.
• γ_t = σ(W_γ · x_t + b_γ) ∈ (0, 1) is the decoupled memory gate controlling damping independently.

Exact Real Dual-Component Parallel Prefix Scan:

S = cumsum(log γ_t).clamp(min=-12.0, max=0.0)
Φ = cumsum(θ_t)
h_t = exp(S) · [ cos(Φ) · cumsum(u_real) - sin(Φ) · cumsum(u_imag) ]

💻 How to Install and Run QU-SSM

Run QU-SSM-130M-MoE directly using standard Hugging Face transformers:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Prannesshkva/QU-SSM-130M-MoE"
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)

prompt = "Once upon a time in a world of state space models,"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=50)
print(tokenizer.decode(output[0]))

🌐 Universal Multimodal Support in QU-SSM

The QU-SSM backbone supports four continuous and discrete sequence interfaces:

❓ Frequently Asked Questions (FAQ)

What makes QU-SSM different from classical real-decay SSMs?

Classical SSMs enforce negative real decay (A ∈ ℝ⁻), creating a dissipative low-pass filter that loses oscillatory and high-frequency phase information (e^(-α·t) → 0). QU-SSM uses dynamic Lie-group rotations over **SO(2