Puro

Why Puro-2B?

Puro-2B is intended to make billion-parameter pretraining inspectable and affordable for smaller research groups. The release covers more than the final weights:

• A canonical 2B base model and intermediate or controlled checkpoints: this repo.
• Materialized pretraining data: .
• Training implementation: .
• Data-processing implementation: .
• Technical report: .

The main recipe combines RTX 5090 infrastructure, blockwise FP8, MuonH with hyperball constraints, proxy-guided data selection, and a curriculum-aware late continuation followed by checkpoint averaging.

The $5,090 Result, Explained

The collection contains multiple checkpoints with different Phase 2 budgets and recipes. The report's approximately $4.4K result is an observed uniform-recipe checkpoint that already exceeds Qwen2-1.5B on the report's 15-task aggregate. It is not the canonical final checkpoint.

The canonical Puro-2B-Base model is the strongest released endpoint. Its production run used 22,514 measured active-training GPU-hours, corresponding to $6,891 under the report's normalized RTX 5090 rental rate.

These figures are accelerator-only reproduction estimates. They exclude data acquisition and preprocessing, proxy and ablation experiments, failed runs, post-training, evaluation, storage, networking, and research labor. They should not be read as the total cost of developing the project.

The scaling-law panel labels points by cumulative reproduction cost. The model catalog below maps those costs to Phase 2 budget fractions. Each fraction applies only to Phase 2 data exposure, while the cost includes the shared Phase 1 run.

Model Details

This is a pretrained base model. It has not been instruction-tuned or preference-aligned and should not be expected to behave like a chat assistant.

Quickstart

Use a Transformers release that supports the Qwen3 configuration:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "thu-pacman/Puro-2B-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

prompt = "The central limit theorem states that"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation

All numbers below come from the same deterministic OpenCompass pipeline in the technical report. The comparison uses pretrained/base checkpoints throughout. Generation tasks use greedy decoding; multiple-choice tasks use fixed token-likelihood ranking. Scores are percentages.

The four math and code tasks are GSM8K, MATH, sanitized-MBPP, and HumanEval. The eleven reasoning and knowledge tasks are MMLU, MMLU-Pro, ARC-Challenge, ARC-Easy, BoolQ, CommonsenseQA, HellaSwag, PIQA, SocialIQA, WinoGrande, and BBH. Each displayed average is an unweighted arithmetic mean.

The Puro Cost Scaling Law fits five single-run Phase 2 uniform-budget points. It is a recipe-specific empirical scale-down relationship, not a universal law. The fit has no uncertainty interval, and the available experiments do not isolate curriculum ordering, constant-LR continuation, and checkpoint averaging as independent causal gains.

Training

Both phases use a sequence length of 4,096, a global batch size of 1,536 sequences, and a micro-batch size of 2. Main Transformer linear-layer GEMMs use blockwise E4M3 FP8; numerically sensitive operations, master weights, and optimizer states remain in BF16 or FP32 as appropriate.

Selected approximately scale-invariant matrix weights are updated by MuonH with hyperball projection and zero weight decay. The remaining parameters use AdamW with weight decay 0.1. The MuonH matrix group applies a 10x multiplier to the shared base l