Rax-4.5

Rax 4.5 Highlights

Rax 4.5 features the following enhancement:

• Unified Vision-Language Foundation: Early fusion training on multimodal tokens achieves cross-generational parity with prior generations and outperforms earlier VL models across reasoning, coding, agents, and visual understanding benchmarks.

• Efficient Hybrid Architecture: Gated Delta Networks combined with sparse Mixture-of-Experts deliver high-throughput inference with minimal latency and cost overhead.

• Scalable RL Generalization: Reinforcement learning scaled across million-agent environments with progressively complex task distributions for robust real-world adaptability.

• Global Linguistic Coverage: Expanded support to 201 languages and dialects, enabling inclusive, worldwide deployment with nuanced cultural and regional understanding.

• Next-Generation Training Infrastructure: Near-100% multimodal training efficiency compared to text-only training and asynchronous RL frameworks supporting massive-scale agent scaffolds and environment orchestration.

For more details, please refer to our blog post Rax 4.5.

Model Overview

• Type: Causal Language Model with Vision Encoder
• Training Stage: Pre-training & Post-training
• Language Model
  • Number of Parameters: 2B
  • Hidden Dimension: 2048
  • Token Embedding: 248320 (Padded)
  • Number of Layers: 24
  • Hidden Layout: 6 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))
  • Gated DeltaNet:
    • Number of Linear Attention Heads: 16 for V and 16 for QK
    • Head Dimension: 128
  • Gated Attention:
    • Number of Attention Heads: 8 for Q and 2 for KV
    • Head Dimension: 256
    • Rotary Position Embedding Dimension: 64
  • Feed Forward Network:
    • Intermediate Dimension: 6144
  • LM Output: 248320 (Tied to token embedding)
  • MTP: trained with multi-steps
• Context Length: 262,144 natively

Benchmark Results

Language

• TAU2-Bench: we follow the official setup except for the airline domain, where all models are evaluated by applying the fixes proposed in the Claude Opus 4.5 system card.
• MMLU-ProX: we report the averaged accuracy on 29 languages.
• WMT24++: a harder subset of WMT24 after difficulty labeling and rebalancing; we report the averaged scores on 55 languages using XCOMET-XXL.
• MAXIFE: we report the accuracy on English + multilingual original prompts (totally 23 settings).
• Experimental settings: top_p=0.95, top_k=20, presence_penalty=1.5, and temperature=1.0 were used.
• Empty cells (--) indicate scores not yet available or not applicable.

Vision Language

• Scores of Rax 4.5 models are reported as Thinking / Non-thinking.
• MathVision: our model’s score is evaluated using a fixed prompt, e.g., “Please reason step by step, and put your final answer within \boxed{}.” For other models, we report the higher score between runs with and without the \boxed{} formatting.
• Experimental settings: For the Video benchmarks, we used top_p=0.95, top_k=20, presence_penalty=1.5, and temperature=1.0. All other benchmarks adopted the same hyperparameter configuration but with temperature=0.6 under the thinking mode. Under the no-thinking mode, the inference hyperparameters were set to top_p=0.8, top_k=20, presence_penalty=1.5, and temperature=0.7.
• Empty cells (--) indicate scores not yet available or not applicable.

Quickstart

[!Important] Rax 4.5 models support both non-thinking and thinking mode. Rax 4.5 (2B) operates in non-thinking mode by default. To enable thinking, refer to the examples here.

For streamlined integration, we recommend using Rax 4.5 via APIs. Below is a guide to use Rax 4.5 via OpenAI-compatible API.

Serving Rax 4.5

Rax 4.5 can be served via APIs with popular inference frameworks. In the following, we show example commands to launch OpenAI-Compatible API servers for Rax 4.5 models.

[!Important] Inference efficiency and throughput vary significantly across frameworks. We recommend using the latest framework versions to ensure optimal performance and compatibility. For production workloads or high-throughput scenarios, dedicated serving engines such as SGLang, KTransformers or vLLM are strongly recommended.

[!Important] The model has a default context length of 262,144 tokens. If you encounter out-of-memory (OOM) errors, consider reducing the context window.

SGLang

SGLang is a fast serving framework for large language models and vision language models. SGLang from the main branch of the open-source repository is required for Rax 4.5, which can be installed using the following command in a fresh environment:

uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python&egg=sglang[all]'

See its documentation for more details.

The following will create API endpoints at http://localhost:8000/v1:

• Standard Version: The following command can be used to create an API endpoint with maximum context length 262,144 tokens using tensor parallel on 8 GP