Motif-3

The model is pretrained on approximately 12.5 trillion tokens spanning web documents, STEM, code, mathematics, multilingual content, and domain-specialized corpora, with additional emphasis on Korean, reasoning-intensive, legal, and financial data. Post-training combines general supervised fine-tuning, six RL-trained specialist teachers, a software-engineering teacher, and Multi-teacher On-Policy Distillation (MOPD) into a single unified model.

Key Features

• 🧠 Fine-grained sparse MoE — 384 routed experts with only 8 activated per token (plus 1 shared expert), providing a large expert pool at limited per-token compute.
• 📏 Native 256K context (262,144 tokens), trained with window-aware context parallelism.
• ⚙️ Novel architecture — GDLA attention, Expert-Specific PolyNorm, modified mHC, and a built-in MTP head enabling self-speculative decoding.
• 🌐 Multilingual & general-purpose, with a strong bytes-per-token tokenizer for English, Korean, code, and math.
• 🎯 Agentic strengths — particularly strong on long-horizon agentic tool use and terminal-based problem solving, with calibrated abstention on hallucination-sensitive evaluations.

2. Model Summary

3. Evaluation Results

For contextual comparison, Motif 3 is compared with strong open-weight models using scores reported on the corresponding benchmark leaderboards. All Motif 3 evaluations were performed with sampling temperature = 1.0, top-p = 0.95, and a maximum sequence length of 262,144 tokens. (*: public dataset only)

Motif 3 performs particularly well on agentic and tool-oriented benchmarks, while maintaining competitive performance across coding, mathematical reasoning, and general knowledge. On AA-Omniscience it pairs its accuracy with one of the highest non-hallucination scores, indicating a favorable balance between answering correctly and abstaining when unsupported.

4. Architecture

[!NOTE] The architecture and distributed training framework used for Motif 3 are available at MotifTechnologies/motif3-training-example.

Motif 3 is a fully in-house design and introduces several custom components (full details in the technical report):

• Grouped Differential Latent Attention (GDLA) — integrates grouped differential attention (asymmetric signal/noise heads with a token-dependent differential coefficient) with the compressed KV latent of Multi-head Latent Attention, plus a query-dependent output gate. Retains the expressive attention dynamics of differential attention while substantially reducing KV-cache requirements.
• Expert-Specific PolyNorm — replaces the SiLU gate with a learned polynomial normalization whose coefficients are learned independently per expert, reducing activation outliers while allowing each expert to specialize.
• Modified manifold-constrained hyper-connections (mHC) — replaces conventional residual additions with a doubly-stochastic (Birkhoff-polytope) mixing of 4 parallel residual streams; the post-mapping multiplier is annealed from 2 → 1 during pretraining to limit activation-outlier accumulation.
• Multi-Token Prediction (MTP) — a 1-layer MTP head (DeepSeek-V3 style) used as an auxiliary pretraining objective and enabling self-speculative decoding at inference.

5. Deployment — vLLM (Recommended)

[!Note]

• Tested on B200 and H200 GPUs.
• The model ships with a built-in MTP head (num_nextn_predict_layers=1), so it supports self-speculative decoding — add --speculative-config as shown below (num_speculative_tokens: 1 is optimal for this model).
• Supports online block-fp8 quantization with --quantization modelopt_blockfp8
• If you encounter any issues, please open an HF issue.

[!Tip] Looking for a smaller footprint? An NVFP4-quantized checkpoint is available at [Motif-Technologies/Motif-3-NVFP4](htt