Model reference · open weights
Yarn-Llama-2-128k is an open-weight language model from NousResearch. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Released by | NousResearch |
|---|---|
| Type | Language models |
| Task | Text gen |
| Context | 128k tokens |
| Runs with | transformers |
| Released | 2023-08-31 |
| Popularity | 3k downloads / month |
| Licence | Unknown |
About
Nous-Yarn-Llama-2-13b-128k is a state-of-the-art language model for long context, further pretrained on long context data for 600 steps. This model is the Flash Attention 2 patched version of the original model: https://huggingface.co/conceptofmind/Yarn-Llama-2-13b-128k
Note that this model requires the Flash Attention library in order to function correctly, see the Model Usage section for installation instructions.
Starting from the base Llama 2 models, this model was further pretrained on a subset of the PG19 dataset, allowing it to effectively utilize up to 128k tokens of context.
The authors would like to thank Stability AI, Carper AI, and Eleuther AI for their generous support of significant computing resources that enabled the training of these models and the completion of this research. We would also like to thank Jonathan Tow and Dakota Mahan directly for their help in advising on the use of the Stability AI compute cluster. Additionally, we would like to thank a16z, and PygmalionAI, for providing resources to run evaluations and experiments on the models.
Install FA2 and Rotary Extensions:
pip install flash-attn --no-build-isolation
pip install git+https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary
There are no specific prompt formats as this is a pretrained base model.
TODO
We plan to continue training when we have more compute and to improve the dataset and/or instruct tune the models in order to improve the long context performance even further.
The model is available for download on HuggingFace.
From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys yarn-llama-2-128k for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (yarn-llama-2-128k below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/chat/completions \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"yarn-llama-2-128k","messages":[{"role":"user","content":"Hello"}]}'
Create an account — your API key is available in the console. 3M free tokens every 30 days with every new account.