Model reference · open weights
pythia-seed3 is an open-weight language model from EleutherAI, listed in the AxForge catalogue. AxForge can bring it up on EU-owned hardware for you on request — with the licence handled where one is required.
About
PolyPythias This model is part of the PolyPythias suite, an extension of the Pythia project providing 45 additional training runs across 5 model sizes with 9 different random seeds each. These models enable systematic study of training stability and reproducibility in language models. Paper PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs Oskar van der Wal, Pietro Lesci, Max Muller-Eberstein, Naomi Saphra, Hailey Schoelkopf, Willem Zuidema, and Stella Biderman. ICLR 2025. Model Details All models were trained on 300B tokens from The Pile. Naming Convention - pythia-{size}m - Original Pythia model (seed 1234) - pythia-{size}m-seed{1-9} - PolyPythias variants with different random seeds - pythia-160m-data-seed{1-3} - 160M models with only data ordering varied (weight init fixed) - pythia-160m-weight-seed{1-3} - 160M models with only weight initialization varied (data order fixed) The decoupled seed variants (data-seed and weight-seed) allow researchers to separately study the effects of data ordering vs. weight initialization. Quick Start Available Checkpoints Each model provides 154 intermediate checkpoints saved as Git branches: To load a specific checkpoint: Training Data All models were trained on The Pile using pre-shuffled data orderings. The shuffled index files for each seed are available at: EleutherAI/pile-preshuffled-seeds This dataset contains .idx files for seeds 0-9 used with MMapIndexedDataset to load the memory-mapped Pile data in the correct order for each seed. Reproducing Training Data Order To reproduce the exact data ordering used for a specific seed: 1. Download the Pile dataset and tokenize it using the Pythia tokenizer 2. Download the corresponding seed folder from pile-preshuffled-seeds: 3. Use the idx files with GPT-NeoX's MMapIndexedDataset: For complete training reproduction instructions, see the Pythia GitHub repository. All PolyPythias Models The complete collection is available at: EleutherAI/polypythias 14M Parameter Models - pythia-14m-seed1 through pythia-14m-seed9 31M Parameter Models - pythia-31m-seed1 through pythia-31m-seed9 70M Parameter Models - pythia-70m-seed1 through pythia-70m-seed9 160M
Summarised from the published model card. Read the full card on the HuggingFace links below.
Specifications
| Maker | EleutherAI |
|---|---|
| Type | Language models |
| Parameters (lead) | 213M |
| Context | 2k tokens |
| Variants | 2 |
| Runs with | transformers |
| Released | 2023-02-15 |
| Popularity | 31k downloads / month |
| Licence | Open weights |
How it works
Variants
Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.
Using it via the API
Once AxForge deploys pythia-seed3 for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (pythia-seed3 below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/chat/completions \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"pythia-seed3","messages":[{"role":"user","content":"Hello"}]}'
Licence
Open weights under apache-2.0 — commercial use is permitted. Deploy it on AxForge EU hardware on request. Read the licence ↗