petitgpt

Compatibility: the released CLI requires a CUDA GPU and does not implement a Transformers AutoModel loading path. GGUF, ONNX, vLLM, llama.cpp, and CPU inference are neither implemented nor validated in this release.

Quick start

Use a CUDA-enabled environment matching requirements-inference-tested.txt; the download command below assumes the Hugging Face CLI is already installed. The environment was Python 3.10.12, PyTorch 2.11.0+cu126, NumPy 2.2.6, tokenizers 0.22.2, and safetensors 0.8.0 on an RTX 4090. See the run guide for environment and input details.

hf download yqi0/petitgpt \
  --revision 7bf3df96e6880b242b2907d1e68093435feacd75 \
  --local-dir ./petitgpt-research-v1

(cd ./petitgpt-research-v1 && sha256sum -c SHA256SUMS)

python ./petitgpt-research-v1/inference.py \
  --model-directory ./petitgpt-research-v1 \
  --prompt "Say hello in one sentence." \
  --profile bf16_native \
  --max-new-tokens 32

The greeting prompt is only an example. Download the complete file set and keep the src/ directory in place—inference.py imports from it. The output is JSON containing the generated token IDs, the raw decoded text (including the terminal EOS token when one was produced), and the stop reason.

The same bundle also supports --profile fp32_math and chat-style input via --messages-json. Decoding is greedy; --max-new-tokens accepts 1–384; and the prompt plus the generation budget must fit within 2,048 tokens. Invalid input or context overflow raises an explicit error rather than being silently repaired or truncated. No default system message is inserted. The two numerical profiles are not claimed to produce identical output.

Model specification

The complete checkpoint-derived settings are in config.json. special_tokens_map.json describes the native token convention; it is not a Transformers loader contract.

Training and model lineage

Pretraining consumed approximately 13B tokens on a single RTX 4090: the exact retained packed count was 13,000,005,634 tokens, and optimizer updates traversed 12,999,720,960 model-input positions. It ran in two stages. Stage A (about 10B tokens) drew on FineWeb-Edu dedup (71.11%), DCLM-Edu (20.32%), FineWiki EN (5.08%), and Python-Edu (3.50%). Stage B (about 3B tokens) added structured tutorial content, PES2O, and StackExchange while retaining the Stage A source families. Stage budgets, source shares, pinned dataset revisions, and accounting definitions are in PRETRAIN_SOURCE_MIXTURE.csv and the technical report.

Ten recorded validation measurements, with no smoothing; lines connect observations within each stage. The Stage B detail panel uses a different vertical scale. See pretraining results for the values and measurement scope.

The released weights are a parameter interpolation between two post-training checkpoints. More precisely, in the released model's training path, P2 is supervised fine-tuning on concise instructions, and P3 is a further adaptation stage on basic instruction tasks with replay examples from earlier instruction data:

Pretrained Base → P2 instruction SFT → P3 step320
alpha075 = P2 + 0.75 × (P3_step320 − P2)

Later experiments explored DPO, response distillation, LoRA, and unified Base-SFT. Some brought local gains alongside losses in other capabilities; none was selected to replace alpha075. These experiments are not ancestors of the released weights. A separate soft-logit distillation lab used only external models. For exact hashes and branch history, consult MODEL_PROVENANCE.json and the technical report. Further detail is in the GitHub repository.

Evaluation

All comparisons below are against SmolLM-135M-Instruct and SmolLM2-135M-Instruct. The picture is mixed rather than uniformly favorable: under the project's fixed protocol, petitgpt-alpha075 leads both baselines on ARC-Easy and ARC-Challenge, trails both on PIQA and HellaSwag, and falls between SmolLM and SmolLM2 on IFEval.

All comparison results below were measured in this project on pinned model revisions; the SmolLM and SmolLM2 numbers are our own evaluations under the protocols described here, not scores copied from their official model cards.

Zero-shot likelihood benchmarks