Model reference · open weights
Trinity-Large-TrueBase is an open-weight language model from arcee-ai. Trinity-Large-TrueBase (BF16) weighs 797 GB; the smallest configuration that runs it is 8× H200 141 GB.
What it is
| Released by | arcee-ai |
|---|---|
| Type | Language models |
| Task | Text gen |
| Parameters (lead) | 398.6B |
| Context | 8,192 tokens |
| Runs with | transformers |
| Released | 2026-01-27 |
| Popularity | 234 downloads / month |
| Weights | 797 GB (Trinity-Large-TrueBase (BF16), file size) |
| Licence | Its own licence terms |
What it runs on
Weights 797 GB (file size) · KV cache 61 MB per 1,000 tokens of context, at 16 bits (vLLM's default for this build; an 8-bit cache halves it) · plus 1.5 GB a request for its sliding-window layers · runtime overhead from 1.5 GB on a small card · context up to 8,192 tokens.
| Card | Requests at once 8K tokens each | Requests at once 32K tokens each | Longest single request | Counted memory |
|---|---|---|---|---|
| RTX 3060 12 GB … 8× A100 80 GB 14 smaller cards | — | — | — | |
| 8× H200 141 GB tensor parallel | 117 | — | all 8K | 138 GB a card |
| 8× B200 180 GB tensor parallel | 255 | — | all 8K | 176 GB a card |
| Requests at once | 8K tokens each | 32K tokens each |
|---|---|---|
| 1 | 801 GB | — |
| 5 | 809 GB | — |
| 8 | 815 GB | — |
| 16 | 831 GB | — |
| 32 | 863 GB | — |
| 64 | 928 GB | — |
On one card, with vLLM's small-card settings (2,048 tokens a step). Cards of 70 GB and more reserve more per request and more overhead — each row above uses its own card's settings.
Estimates, not measurements, checked against published vLLM startup logs. The weights are the build's file size; the cache is calculated from its config (attention with sliding-window layers); the overhead is an estimate of vLLM's own memory with that card's default settings. "Requests at once" is how many requests of that length vLLM admits — its reservation at full length, with --max-model-len set to that length; requests that stay shorter fit more. "Longest single request" is the most one request can hold there: below the model's maximum, vLLM starts only with --max-model-len set at or under it. "Counted memory" is vLLM's default 92 % of what CUDA reports for the card (the DGX Spark: about 100 GiB of its shared 128 GB). A tensor-parallel split pools the cards' memory and speeds each token up, at the cost of the link between them; a layer split (llama.cpp) holds more but does not make one request faster. Assumes vLLM 0.10 or later.
From the model card
src="https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/i-v1KyAMOW_mgVGeic9WJ.png" alt="Arcee Trinity Large" style="max-width: 100%; height: auto;" >
Trinity-Large-TrueBase is a base pretraining checkpoint from Arcee AI's Trinity Large training run. It is a 398B-parameter sparse Mixture-of-Experts (MoE) model with approximately 13B active parameters per token. The checkpoint was captured after 10 trillion tokens of pretraining, prior to learning-rate annealing and before any instruction tuning or reinforcement learning.
This checkpoint is intended for research, probing, ablation studies, and downstream fine-tuning and comes without any pre-baked alignment, instruction formatting, or preference optimization.
More details on the training of Trinity Large are available in the technical report.
The Trinity Large family consists of three checkpoints from the same training run:
Trinity-Large-TrueBase uses a sparse MoE configuration designed to maximize efficiency while maintaining large-scale capacity.
| Hyperparameter | Value |
|---|---|
| Total parameters | ~398B |
| Active parameters per token | ~13B |
| Experts | 256 |
| Active experts | 4 |
| Routing strategy | 4-of-256 (1.56% sparsity) |
| Dense layers | 6 |
| Pretraining context length | 8,192 |
| Architecture | Sparse MoE (AfmoeForCausalLM) |
Note: Extended context support (e.g., 512k) was introduced after this checkpoint and is not available in TrueBase.
| Benchmark | N-shot | Metric | Score | Stderr |
|---|---|---|---|---|
| arc_challenge_0shot | 0 | acc_norm,none | 0.6237 | ±0.0142 |
| bbh_fewshot | 3 | exact_match,remove_whitespace | 0.5784 | ±0.0054 |
| gpqa_diamond_5shot | 5 | acc_norm,none | 0.4091 | ±0.0350 |
| gpqa_diamond_generative_5shot | 5 | exact_match,flexible-extract | 0.3788 | ±0.0346 |
| gsm8k_8shot | 8 | exact_match,flexible-extract | 0.8036 | ±0.0109 |
| gsm8k_cot | 8 | exact_match,flexible-extract | 0.8044 | ±0.0109 |
| hellaswag_5shot | 5 | acc_norm,none | 0.8813 | ±0.0032 |
| humaneval_plus | 0 | pass@1,create_test | 0.5183 | ±0.0391 |
| leaderboard_math_hard | 4 | exact_match,none | 0.2696 | ±0.0113 |
| mbpp_plus | 3 | pass_at_1,none | 0.8095 | ±0.0202 |
| minerva_math500 | 4 | math_verify,none | 0.4820 | ±0.0224 |
| mmlu_5shot | 5 | acc,none | 0.7845 | ±0.0033 |
| mmlu_generative_5shot | 5 | exact_match,get_response | 0.7848 | ±0.0033 |
| mmlu_pro | 5 | exact_match,custom-extract | 0.5160 | ±0.0044 |
| triviaqa_5shot | 5 | exact_match,remove_whitespace | 0.8096 | ±0.0029 |
| winogrande_5shot | 5 | acc,none | 0.8145 | ±0.0109 |
This checkpoint branches from the main Trinity Large run at the 10T-token mark, prior to learning-rate decay or post-training phases.
Optimizer learning rates after WSD warm-up:
Muon was used to support larger critical batch sizes in a highly sparse MoE regime.
Most base model releases include instruction data, annealed training dynamics, or early alignment stages. Trinity-Large-TrueBase excludes these, providing an opportunity to study what large-scale models learn from pretraining data alone. This checkpoint is intended as a foundation for research rather than as a finished conversational assistant.
Trinity-Large-TrueBase is released under the OpenMDW License, version 1.1 (OpenMDW-1.1).
Quoted from the model card on Hugging Face — the full card is behind the Hugging Face link above.