pythia-data-seed1

About

What pythia-data-seed1 is

PolyPythias This model is part of the PolyPythias suite, an extension of the Pythia project providing 45 additional training runs across 5 model sizes with 9 different random seeds each. These models enable systematic study of training stability and reproducibility in language models. Paper PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs Oskar van der Wal, Pietro Lesci, Max Muller-Eberstein, Naomi Saphra, Hailey Schoelkopf, Willem Zuidema, and Stella Biderman. ICLR 2025. Model Details All models were trained on 300B tokens from The Pile. Naming Convention - pythia-{size}m - Original Pythia model (seed 1234) - pythia-{size}m-seed{1-9} - PolyPythias variants with different random seeds - pythia-160m-data-seed{1-3} - 160M models with only data ordering varied (weight init fixed) - pythia-160m-weight-seed{1-3} - 160M models with only weight initialization varied (data order fixed) The decoupled seed variants (data-seed and weight-seed) allow researchers to separately study the effects of data ordering vs. weight initialization. Quick Start Available Checkpoints Each model provides 154 intermediate checkpoints saved as Git branches: To load a specific checkpoint: Training Data All models were trained on The Pile using pre-shuffled data orderings. The shuffled index files for each seed are available at: EleutherAI/pile-preshuffled-seeds This dataset contains .idx files for seeds 0-9 used with MMapIndexedDataset to load the memory-mapped Pile data in the correct order for each seed. Reproducing Training Data Order To reproduce the exact data ordering used for a specific seed: 1. Download the Pile dataset and tokenize it using the Pythia tokenizer 2. Download the corresponding seed folder from pile-preshuffled-seeds: 3. Use the idx files with GPT-NeoX's MMapIndexedDataset: For complete training reproduction instructions, see the Pythia GitHub repository. All PolyPythias Models The complete collection is available at: EleutherAI/polypythias 14M Parameter Models - pythia-14m-seed1 through pythia-14m-seed9 31M Parameter Models - pythia-31m-seed1 through pythia-31m-seed9 70M Parameter Models - pythia-70m-seed1 through pythia-70m-seed9 160M

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys pythia-data-seed1 for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (pythia-data-seed1 below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"pythia-data-seed1","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Papers

Licence

Open weights

Sources

Weights & code

Explore

More language models