Surjo

Model Specifications

Evaluation Results

Architecture Details

Surjo Gen-3 combines fast recurrent sequence processing with full-attention boundary layers:

• Prelude (Layer 0): Standard attention with subtractive value projection (XSA) to anchor context.
• Recurrent Core (Layers 1–8, tied 2× passes): Two interleaved groups consisting of 3× GDN-2 linear attention layers followed by 1× XSA full attention layer. GDN-2 uses chunked/fused recurrent kernels via Flash Linear Attention (FLA).
• Coda (Layer 9): Terminal full-attention XSA layer.
• Tied Head: The output projection shares weights with the 32,768-token embedding layer.

Pretraining

• Dataset (20B tokens):
  • 60% FinePhrase
  • 20% DCLM
  • 10% FineMath
  • 10% Cornstack Python
• Training Schedule: 20,000 steps (~1M tokens/step) at a sequence length of 1024 tokens.
• Optimizer: Hybrid Muon (peak lr = 0.02) and AdamW (peak lr = 0.0036).

Usage

Requirements

pip install torch transformers accelerate
pip install -U git+https://github.com/fla-org/flash-linear-attention

Generation Example

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SurjoLabs/Surjo-50m"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "The secret to scaling small language models is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation & License

This project is released under the Apache 2.0 License.

@software{surjo50m2026,
  author = {SurjoLabs},
  title = {Surjo-50M: Hybrid Recurrent Language Model},
  year = {2026},
  url = {https://huggingface.co/SurjoLabs/Surjo-50m}
}