Olmo-3-1025

Model Details

Model Card for Olmo 3 7B

We introduce Olmo 3, a new family of 7B and 32B models. This suite includes Base, Instruct, and Think variants. The Base models were trained using a staged training approach.

Olmo is a series of Open language models designed to enable the science of language models. These models are trained on the Dolma 3 dataset. We are releasing all code, checkpoints, and associated training details.

The core models released in this batch include the following:

Installation

Olmo 3 is supported in transformers v4.57.0 or higher:

pip install transformers>=4.57.0

Inference

You can use OLMo with the standard HuggingFace transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer
olmo = AutoModelForCausalLM.from_pretrained("allenai/Olmo-3-1025-7B")
tokenizer = AutoTokenizer.from_pretrained("allenai/Olmo-3-1025-7B")
message = ["Language modeling is "]
inputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)
# optional verifying cuda
# inputs = {k: v.to('cuda') for k,v in inputs.items()}
# olmo = olmo.to('cuda')
response = olmo.generate(**inputs, max_new_tokens=100, do_sample=True, top_k=0, temperature=1.0, top_p=0.7)
print(tokenizer.batch_decode(response, skip_special_tokens=True)[0])
>> 'Language modeling is  a key component of any text-based application, but its effectiveness...'

For faster performance, you can quantize the model using the following method:

AutoModelForCausalLM.from_pretrained("allenai/Olmo-3-1025-7B",
    torch_dtype=torch.float16,
    load_in_8bit=True)  # Requires bitsandbytes

The quantized model is more sensitive to data types and CUDA operations. To avoid potential issues, it's recommended to pass the inputs directly to CUDA using:

inputs.input_ids.to('cuda')

We have released checkpoints for these models. For pretraining, the naming convention is stage1-stepXXX. The conventions for midtraining and long context are stage2-stepXXX and stage3-stepXXX, respectively.

To load a specific model revision with HuggingFace, simply add the argument revision:

olmo = AutoModelForCausalLM.from_pretrained("allenai/Olmo-3-1025-7B", revision="stage1-step10000")

Or, you can access all the revisions for the models via the following code snippet:

from huggingface_hub import list_repo_refs
out = list_repo_refs("allenai/Olmo-3-1025-7B")
branches = [b.name for b in out.branches]

Fine-tuning

Model fine-tuning can be done from the final checkpoint (the main revision of this model) or many intermediate checkpoints. Two recipes for tuning are available.

1. Fine-tune with the OLMo-core repository:

torchrun --nproc-per-node=8 ./src/scripts/official/OLMo3/OLMo-3-1025-7B-pretrain-1.py run01

You can override most configuration options from the command-line. For example, to override the learning rate you could launch the script like this:

torchrun --nproc-per-node=8 ./src/scripts/official/OLMo3/OLMo-3-1025-7B-pretrain-1.py run01 --train_module.optim.lr=3e-4

For more documentation, see the GitHub readme.

Model Description

• Developed by: Allen Institute for AI (Ai2)
• Model type: a Transformer style autoregressive language model.
• Language(s) (NLP): English
• License: The code and model are released under Apache 2.0.
• Contact: Technical inquiries: olmo@allenai.org. Press: press@allenai.org
• Date cutoff: Dec 2024

Model Sources

• Project Page: https://allenai.org/olmo
• Repositories:
  • Core repo (training, inference, fine-tuning etc.): https://github.com/allenai/OLMo-core
  • Evaluation code: https://github.com/allenai/OLMo-Eval
  • Further fine-tuning code: https://github.com/allenai/open-instruct
• W&B Report: https://wandb.ai/ai2-llm/Olmo-3-1025-7B/reports/Olmo-3-7B-October-2025--VmlldzoxNDcwOTM0NA
• Paper: https://allenai.org/papers/olmo3

Evaluation

Core model results for Olmo 3 7B are found below.

| Model | Olmo 3-Eval Math | BigCodeBench | HumanEval | DeepSeek LeetCode | DS 1000 | MBPP | MultiPL HumanEval | MultiPL MBPPP | Olmo 3-Eval Code | ARC MC | MMLU STEM | MedMCQA MC | MedQA MC | SciQ MC | Olmo 3-Eval MC_STEM | MMLU Humanities | MMLU Social Sci. | MMLU Other | CSQA MC | PIQA MC | SocialIQA MC | CoQA Gen2M