OLMo-0724

Model Details

The core models released in this batch are the following:

[Coming soon] We are releasing many checkpoints for these models, for every 1000 training steps. The naming convention is stepXXX-tokensYYYB.

To load a specific model revision with HuggingFace, simply add the argument revision:

olmo = AutoModelForCausalLM.from_pretrained("allenai/OLMo-1B-0724-hf", revision="step1000-tokens4B")

All revisions/branches are listed in the file revisions.txt. Or, you can access all the revisions for the models via the following code snippet:

from huggingface_hub import list_repo_refs
out = list_repo_refs("allenai/OLMo-1B-0724-hf")
branches = [b.name for b in out.branches]

Model Description

• Developed by: Allen Institute for AI (AI2)
• Supported by: Databricks, Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University, AMD, CSC (Lumi Supercomputer), UW
• Model type: a Transformer style autoregressive language model.
• Language(s) (NLP): English
• License: The code and model are released under Apache 2.0.
• Contact: Technical inquiries: olmo at allenai dot org. Press: press at allenai dot org
• Date cutoff: Oct. 2023, with most data from Feb./March 2023 based on Dolma dataset version.

Model Sources

• Project Page: https://allenai.org/olmo
• Repositories:
  • Core repo (training, inference, fine-tuning etc.): https://github.com/allenai/OLMo
  • Evaluation code: https://github.com/allenai/OLMo-Eval
  • Further fine-tuning code: https://github.com/allenai/open-instruct
• Paper: Link

Uses

Inference

Install Transformers. Then proceed as usual with HuggingFace:

from transformers import AutoModelForCausalLM, AutoTokenizer
olmo = AutoModelForCausalLM.from_pretrained("allenai/OLMo-1B-0724-hf")
tokenizer = AutoTokenizer.from_pretrained("allenai/OLMo-1B-0724-hf")
message = ["Language modeling is "]
inputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)
# optional verifying cuda
# inputs = {k: v.to('cuda') for k,v in inputs.items()}
# olmo = olmo.to('cuda')
response = olmo.generate(**inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)
print(tokenizer.batch_decode(response, skip_special_tokens=True)[0])
>> 'Language modeling is the first step to build natural language generation...'

Alternatively, with the pipeline abstraction:

from transformers import pipeline
olmo_pipe = pipeline("text-generation", model="allenai/OLMo-1B-0724-hf")
print(olmo_pipe("Language modeling is "))
>> 'Language modeling is a branch of natural language processing that aims to...'

Or, you can make this slightly faster by quantizing the model, e.g. AutoModelForCausalLM.from_pretrained("allenai/OLMo-1B-0724-hf", torch_dtype=torch.float16, load_in_8bit=True) (requires bitsandbytes). The quantized model is more sensitive to typing / cuda, so it is recommended to pass the inputs as inputs.input_ids.to('cuda') to avoid potential issues.

Fine-tuning

Model fine-tuning can be done from the final checkpoint (the main revision of this model) or many intermediate checkpoints. Two recipes for tuning are available.

1. Fine-tune with the OLMo repository:

torchrun --nproc_per_node=8 scripts/train.py {path_to_train_config} \
    --data.paths=[{path_to_data}/input_ids.npy] \
    --data.label_mask_paths=[{path_to_data}/label_mask.npy] \
    --load_path={path_to_checkpoint} \
    --reset_trainer_state

For more documentation, see the GitHub readme.

2. Further fine-tuning support is being developing in AI2's Open Instruct repository. Details are here.

Evaluation

Core model results for the new and original 7B model are found below.