Model reference · open weights
cRia-LM is an open-weight language model from sz14. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Released by | sz14 |
|---|---|
| Type | Language models |
| Task | Text gen |
| Parameters (lead) | 76M |
| Context | 4k tokens |
| Runs with | transformers |
| Released | 2026-08-20 |
| Popularity | 3k downloads / month |
| Licence | Open weights |
About
NOTE: A stale checkpoint was accidentally pushed, please don't use till fixed
Cria (noun): a baby llama, alpaca, vicuña, or guanaco. Pronounced ˈkrē-ə.
~ Merriam-Webster
cRia-LM-75M is a 75.7M-parameter base language model built as a Relaxed Recursive Transformer (RRT). It uses a shared 11-layer recurrent block evaluated twice, with pass-specific LoRA parameters on the second traversal.
Training was carried out in three stages. Stage 1 established the 2K base model over 10B tokens. Stage 2 continued training with a 2B-token budget and a capability-focused data curriculum; the released Stage 2 checkpoint is step 10,000, corresponding to about 1.31B continuation tokens. Stage 3 extended the context window from 2,048 to 4,096 tokens with a 50M-token run on codelion/sutra-1B; the released checkpoint was selected at roughly 5M tokens.
This is a base model. It has not been instruction tuned and does not use a chat template. The Transformer layer design follows HuggingFaceTB/SmolLM2-135M, with QK-Norm added for training stability.
| Specification | Value |
|---|---|
| Parameters | 75.7M |
| Architecture | Relaxed Recursive Transformer |
| Unique Transformer layers | 13 |
| Effective depth | 24 |
| Prelude layers | 1 |
| Shared recurrent layers | 11 |
| Recurrent passes | 2 |
| Coda layers | 1 |
| Hidden size | 576 |
| MLP intermediate size | 1,536 |
| Attention heads | 9 query heads |
| KV heads | 3 |
| Attention type | Grouped-query attention |
| Head dimension | 64 |
| MLP | SwiGLU |
| Normalization | RMSNorm |
| Attention normalization | QK-Norm |
| Position encoding | RoPE |
| RoPE theta | 100,000 |
| Context length | 4,096 tokens |
| Vocabulary size | 49,152 |
| Tokenizer | cRia-LM-75M BPE tokenizer, per-digit tokenized |
| Token embedding | Tied, factorized |
| Embedding rank | 210 |
| Recurrent LoRA rank | 172 |
| KV cache | Recursion-wise, one slot per effective layer/pass |
| Model type | Base causal language model |
cRia-LM-75M uses 13 unique Transformer layers arranged as:
1 prelude + (11 shared layers x 2 recurrent passes) + 1 coda
This gives an effective depth of 24 Transformer layers while storing the main parameters for only 13 unique layers.
The 11-layer recurrent block is shared across both passes. On the second pass, the recurrent linear projections receive rank-172 LoRA updates. The two traversals therefore share the base weights while retaining pass-specific capacity.
Each Transformer layer uses:
hidden size 576
9 query heads and 3 KV heads with grouped-query attention
head dimension 64
SwiGLU feed-forward network with intermediate size 1,536
RMSNorm
rotary position embeddings
QK-Norm
src="https://hfviewer.com/api/card.svg?source=sz14%2FcRia-LM-75M&granularity=auto&v=20260516-title-pills-card" alt="Architecture graph for sz14/cRia-LM-75M. Open in hfviewer" width="100%" />
The input embedding and language-model readout are tied through a rank-210 factorization:
49,152 x 210
210 x 576
The same factors are used for token lookup and output projection. This keeps the full 49,152-token vocabulary while reducing the parameter cost of the embedding and output head.
Stage 1 trained the model for 10B tokens at a sequence length of 2,048. The objective combined next-token cross-entropy with logit-level knowledge distillation from HuggingFaceTB/SmolLM2-360M, with dynamic CE/KD scale balancing.
The Stage 1 sampling mix was:
| Data source | Sampling weight |
|---|---|
| FineWeb-Edu | 48% |
| DCLM-Edu | 32% |
| Cosmopedia-v2 | 12% |
| FineMath-4+ | 5% |
| StarCoder Python | 3% |
Stage 1 used Muon for matrix parameters and AdamW for the remaining parameter groups. The learning-rate schedule was warmup-stable-decay: 1% warmup, decay beginning at 80% of the run, and the final 20% used for decay.
Stage 2 used a 2B-token training budget at 2,048-token context. The released Stage 2 checkpoint is step 10,000, or 1,310,720,000 tokens at 131,072 tokens per optimizer step.
The phase used two-teacher distillation: same-tokenizer KL from HuggingFaceTB/SmolLM2-360M and cross-tokenizer supervision from Qwen/Qwen3.5-0.8B-Base using X-Token. The run used the H-KL cross-tokenizer objective after the tokenizer-coverage audit, with dynamic CE/KD balancing retained around the combined distillation loss.
Phase 2A is the capability-injection mix. It remains active through 65% of the planned Stage 2 run. The original configuration assigned 8% to Nemotron QA, but that source was unavailable when the run started. The loader dropped it and renormalized the remaining weights; the table below shows the effective mix that was actually sampled.
| Data source | Effective sampling weight |
|---|---|
| FineWeb-Edu | 19.57% |
| DCLM-Edu | 13.04% |
| Cosmopedia-v2 | 10.87% |
| Cosmopedia Stories | 8.70% |
| Cosmopedia WikiHow | 7.61% |
| Cosmopedia Science | 9.78% |
| Wikipedia | 10.87% |
| FineMath | 5.43% |
| Nemotron Math | 3.26% |
| InfiWebMath | 3.26% |
| StarCoder Python | 4.35% |
| FineWeb-HQ | 3.26% |
Phase 2B is the cooldown mix. It removes the noisier sources and shifts more weight toward higher-quality web, educational, encyclopedic, science, and math data.
| Data source | Sampling weight |
|---|---|
| FineWeb-HQ | 18% |
| Cosmopedia-v2 | 14% |
| Cosmopedia Science | 14% |
| Wikipedia | 14% |
| FineMath | 12% |
| Cosmopedia WikiHow | 8% |
| Cosmopedia Stories | 8% |
| DCLM-Edu | 8% |
| StarCoder Python | 4% |
Stage 1 had already completed a full decay, so Stage 2 re-warmed to a lower peak rather than returning to the original Stage 1 learning rate.
| Setting | Value |
|---|---|
| Stage 2 token bud |
From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys cria-lm for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (cria-lm below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/chat/completions \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"cria-lm","messages":[{"role":"user","content":"Hello"}]}'
Create an account — your API key is available in the console. 3M free tokens every 30 days with every new account.