Model reference · open weights
OpenELM is an open-weight language model from apple. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Released by | apple |
|---|---|
| Type | Language models |
| Task | Text gen |
| Parameters (lead) | 3.0B |
| Runs with | transformers |
| Released | 2024-04-12 |
| Popularity | 610 downloads / month |
| Licence | Commercial licence needed |
About
Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, Mohammad Rastegari
We introduce OpenELM, a family of Open Efficient Language Models. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. We pretrained OpenELM models using the CoreNet library. We release both pretrained and instruction tuned models with 270M, 450M, 1.1B and 3B parameters. We release the complete framework, encompassing data preparation, training, fine-tuning, and evaluation procedures, alongside multiple pre-trained checkpoints and training logs, to facilitate open research.
Our pre-training dataset contains RefinedWeb, deduplicated PILE, a subset of RedPajama, and a subset of Dolma v1.6, totaling approximately 1.8 trillion tokens. Please check license agreements and terms of these datasets before using them.
We have provided an example function to generate output from OpenELM models loaded via HuggingFace Hub in generate_openelm.py.
You can try the model by running the following command:
python generate_openelm.py --model apple/OpenELM-3B-Instruct --hf_access_token [HF_ACCESS_TOKEN] --prompt 'Once upon a time there was' --generate_kwargs repetition_penalty=1.2
Please refer to this link to obtain your hugging face access token.
Additional arguments to the hugging face generate function can be passed via generate_kwargs. As an example, to speedup the inference, you can try lookup token speculative generation by passing the prompt_lookup_num_tokens argument as follows:
python generate_openelm.py --model apple/OpenELM-3B-Instruct --hf_access_token [HF_ACCESS_TOKEN] --prompt 'Once upon a time there was' --generate_kwargs repetition_penalty=1.2 prompt_lookup_num_tokens=10
Alternatively, try model-wise speculative generation with an assistive model by passing a smaller model through the assistant_model argument, for example:
python generate_openelm.py --model apple/OpenELM-3B-Instruct --hf_access_token [HF_ACCESS_TOKEN] --prompt 'Once upon a time there was' --generate_kwargs repetition_penalty=1.2 --assistant_model [SMALLER_MODEL]
| Model Size | ARC-c | ARC-e | BoolQ | HellaSwag | PIQA | SciQ | WinoGrande | Average |
|---|---|---|---|---|---|---|---|---|
| OpenELM-270M | 26.45 | 45.08 | 53.98 | 46.71 | 69.75 | 84.70 | 53.91 | 54.37 |
| OpenELM-270M-Instruct | 30.55 | 46.68 | 48.56 | 52.07 | 70.78 | 84.40 | 52.72 | 55.11 |
| OpenELM-450M | 27.56 | 48.06 | 55.78 | 53.97 | 72.31 | 87.20 | 58.01 | 57.56 |
| OpenELM-450M-Instruct | 30.38 | 50.00 | 60.37 | 59.34 | 72.63 | 88.00 | 58.96 | 59.95 |
| OpenELM-1_1B | 32.34 | 55.43 | 63.58 | 64.81 | 75.57 | 90.60 | 61.72 | 63.44 |
| OpenELM-1_1B-Instruct | 37.97 | 52.23 | 70.00 | 71.20 | 75.03 | 89.30 | 62.75 | 65.50 |
| OpenELM-3B | 35.58 | 59.89 | 67.40 | 72.44 | 78.24 | 92.70 | 65.51 | 67.39 |
| OpenELM-3B-Instruct | 39.42 | 61.74 | 68.17 | 76.36 | 79.00 | 92.50 | 66.85 | 69.15 |
| Model Size | ARC-c | HellaSwag | MMLU | TruthfulQA | WinoGrande | Average |
|---|---|---|---|---|---|---|
| OpenELM-270M | 27.65 | 47.15 | 25.72 | 39.24 | 53.83 | 38.72 |
| OpenELM-270M-Instruct | 32.51 | 51.58 | 26.70 | 38.72 | 53.20 | 40.54 |
| OpenELM-450M | 30.20 | 53.86 | 26.01 | 40.18 | 57.22 | 41.50 |
| OpenELM-450M-Instruct | 33.53 | 59.31 | 25.41 | 40.48 | 58.33 | 43.41 |
| OpenELM-1_1B | 36.69 | 65.71 | 27.05 | 36.98 | 63.22 | 45.93 |
| OpenELM-1_1B-Instruct | 41.55 | 71.83 | 25.65 | 45.95 | 64.72 | 49.94 |
| OpenELM-3B | 42.24 | 73.28 | 26.76 | 34.98 | 67.25 | 48.90 |
| [OpenELM-3B-Instruct](https |
From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys openelm for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (openelm below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/chat/completions \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"openelm","messages":[{"role":"user","content":"Hello"}]}'
Create an account — your API key is available in the console. 3M free tokens every 30 days with every new account.