Model reference · open weights
OLMoE-0924 is an open-weight language model from allenai, listed in the AxForge catalogue. AxForge can bring it up on EU-owned hardware for you on request — with the licence handled where one is required.
About
Model Summary OLMoE-1B-7B is a Mixture-of-Experts LLM with 1B active and 7B total parameters released in September 2024 (0924). It yields state-of-the-art performance among models with a similar cost (1B) and is competitive with much larger models like Llama2-13B. OLMoE is 100% open-source. This information and more can also be found on the OLMoE GitHub repository. - Paper: https://arxiv.org/abs/2409.02060 - Pretraining Checkpoints, Code, Data and Logs. - SFT (Supervised Fine-Tuning) Checkpoints, Code, Data and Logs. - DPO/KTO (Direct Preference Optimization/Kahneman-Tversky Optimization), Checkpoints, Preference Data, DPO code, KTO code and Logs. Use Install transformers from source until a release after this PR & torch and run: You can list all revisions/branches by installing huggingface-hub & running: Important branches: - step1200000-tokens5033B: Pretraining checkpoint used for annealing. There are a few more checkpoints after this one but we did not use them. - main: Checkpoint annealed from step1200000-tokens5033B for an additional 100B tokens (23,842 steps). We use this checkpoint for our adaptation (https://huggingface.co/allenai/OLMoE-1B-7B-0924-SFT & https://huggingface.co/allenai/OLMoE-1B-7B-0924-Instruct). - fp32: FP32 version of main. The model weights were stored in FP32 during training but we did not observe any performance drop from casting them to BF16 after training so we upload all weights in BF16. If you want the original FP32 checkpoint for main you can use this one. You will find that it yields slightly different results but should perform around the same on benchmarks. Evaluation Snapshot Citation
Summarised from the published model card. Read the full card on the HuggingFace links below.
Specifications
| Maker | allenai |
|---|---|
| Type | Language models |
| Parameters (lead) | 6.9B |
| Context | 4k tokens |
| Variants | 1 |
| Runs with | transformers |
| Released | 2024-07-20 |
| Popularity | 155k downloads / month |
| Likes | 148 |
| Licence | Open weights |
How it works
Variants
Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.
| Variant | Params | Precision | VRAM | Fits 16 GB | Weights |
|---|---|---|---|---|---|
| OLMoE-1B-7B-0924 | 6.9B | BF16 | ~15.9 GB | ✓ | Weights ↗ |
Using it via the API
Once AxForge deploys olmoe-0924 for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (olmoe-0924 below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/chat/completions \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"olmoe-0924","messages":[{"role":"user","content":"Hello"}]}'
Licence
Open weights under apache-2.0 — commercial use is permitted. Deploy it on AxForge EU hardware on request. Read the licence ↗