pythia-deduped

About

What pythia-deduped is

The Pythia Scaling Suite is a collection of models developed to facilitate interpretability research (see paper). It contains two sets of eight models of sizes 70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, and 12B. For each size, there are two models: one trained on the Pile, and one trained on the Pile after the dataset has been globally deduplicated. All 8 model sizes are trained on the exact same data, in the exact same order. We also provide 154 intermediate checkpoints per model, hosted on Hugging Face as branches. The Pythia model suite was designed to promote scientific research on large language models, especially interpretability research. Despite not centering downstream performance as a design goal, we find the models <a href="#evaluations"match or exceed</a the performance of similar and same-sized models, such as those in the OPT and GPT-Neo suites. Previously, we released an early version of the Pythia suite to the public. However, we decided to retrain the model suite to address a few hyperparameter discrepancies. This model card <a href="#changelog"lists the changes</a; see appendix B in the Pythia paper for further discussion. We found no difference in benchmark performance between the two Pythia versions. The old models are still available, but we suggest the retrained suite if you are just starting to use Pythia.<br This is the current release. Please note that all models in the Pythia suite were renamed in January 2023. For clarity, a <a href="#naming-convention-and-parameter-count"table comparing the old and new names</a is provided in this model card, together with exact parameter counts. Pythia-70M-deduped Model Details - Developed by: EleutherAI - Model type: Transformer-based Language Model - Language: English - Learn more: Pythia's GitHub repository for training procedure, config files, and details on how to use. See paper for more evals and implementation details. - Library: GPT-NeoX - License: Apache 2.0 - Contact: to ask questions about this model, join the [EleutherAI Discord](https://discord.gg/zBGx3azzUn), and post them in #release-discussion. Please read the existing Pythia documentation before asking about it in the EleutherAI Discord.

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys pythia-deduped for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (pythia-deduped below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"pythia-deduped","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Papers

Licence

Open weights

Sources

Weights & code

Explore

More language models