Model reference · open weights
K2 is an open-weight language model from IFM. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Released by | IFM |
|---|---|
| Type | Language models |
| Task | Text gen |
| Parameters (lead) | 65.3B |
| Context | 8k tokens |
| Runs with | transformers |
| Released | 2024-04-17 |
| Popularity | 599 downloads / month |
| Licence | Open weights |
About
LLM360 demystifies the training recipe used for Llama 2 70B with K2. K2 is fully transparent, meaning we’ve open-sourced all artifacts, including code, data, model checkpoints, intermediate results, and more.
K2 was developed as a collaboration between MBZUAI, Petuum, and LLM360.
The LLM360 Performance and Evaluation Collection is a robust evaluations set consisting of general and domain specific evaluations to assess model knowledge and function.
Evaluations include standard best practice benchmarks, medical, math, and coding knowledge. More about the evaluations can be found here.
Detailed analysis can be found on the K2 Weights and Biases project here
| Evaluation | Score | Raw Score |
|---|---|---|
| IFEval | 22.52 | 23 |
| BBH | 28.22 | 50 |
| Math Lvl 5 | 2.04 | 2 |
| GPQA | 3.58 | 28 |
| MUSR | 8.55 | 40 |
| MMLU-PRO | 22.27 | 30 |
| Average | 14.53 | 35.17 |
The K2 gallery allows one to browse the output of various prompts on intermediate K2 checkpoints, which provides an intuitive understanding on how the model develops and improves over time. This is inspired by The Bloom Book.
The following data mix was used to train K2 and achieve results in line with Llama 2 70B.
The full data sequence can be found here
| Dataset | Starting Tokens | Multiplier | Total Tokens | % of Total |
|---|---|---|---|---|
| dm-math | 4.33B | 3x | 13B | 1% |
| pubmed-abstracts | 4.77B | 3x | 14.3B | 1.1% |
| uspto | 4.77B | 3x | 14.3B | 1.1% |
| pubmed-central | 26B | 1x | 26B | 2% |
| redpajama.arxiv | 27.3B | 1x | 27.3B | 2.1% |
| starcoder.spm | 67.6B | 0.5x | 33.8B | 2.6% |
| starcoder.fim | 67.6B | 0.5x | 33.8B | 2.6% |
| redpajama.stackexchange | 61.1B | 1x | 61.1B | 4.7% |
| starcoder | 132.6B | 0.5x | 66.3B | 5.1% |
| pile-of-law | 76.7B | 1x | 76.7B | 5.9% |
| redpajama.book | 80.6B | 1x | 80.6B | 6.2% |
| s2orc | 107.9B | 1x | 107.9B | 8.3% |
| redpajama.wikipedia | 22.1B | 6x | 132.6B | 10.2% |
| refinedweb | 612.3B | 1x | 612.3B | 47.1% |
| Totals | - | - | 1.3T | 100% |
| Checkpoints | |
|---|---|
| Checkpoint 380 | Checkpoint 375 |
| Checkpoint 379 | Checkpoint 374 |
| Checkpoint 378 | Checkpoint 373 |
| Checkpoint 377 | Checkpoint 372 |
| Checkpoint 376 | Checkpoint 371 |
| Checkpoints | |
|---|---|
| Checkpoint 360 | Checkpoint 355 |
| Checkpoint 359 | Checkpoint 354 |
| Checkpoint 358 | Checkpoint 353 |
| Checkpoint 357 | Checkpoint 352 |
| Checkpoint 356 | Checkpoint 351 |
[to find all branches: git branch -a]
We provide step-by-step reproducation tutorials for tech enthusiasts, AI practitioners and academic or industry researchers who want to learn pretraining techniques here.
We provide step-by-step finetuning tutorials for tech enthusiasts, AI practitioners and academic or industry researchers [here](https://www.llm360.ai/develo
From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys k2 for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (k2 below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/chat/completions \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"k2","messages":[{"role":"user","content":"Hello"}]}'
Create an account — your API key is available in the console. 3M free tokens every 30 days with every new account.