kai-os_Grug

About

What kai-os_Grug is

Llamacpp imatrix Quantizations of Grug-12B by kai-os Using <a href="https://github.com/ggml-org/llama.cpp/"llama.cpp</a release <a href="https://github.com/ggml-org/llama.cpp/releases/tag/b10068"b10068</a for quantization. Original model: https://huggingface.co/kai-os/Grug-12B All quants made using imatrix option with dataset from here Run them in your choice of tools: - llama.cpp - ramalama - LM Studio - koboldcpp - Jan AI - Text Generation Web UI - LoLLMs - Atomic Chat Note: if it's a newly supported model, you may need to wait for an update from the developers. Prompt format Download a file (not the whole branch) from below: Embed/output weights Some of these quants (Q3KXL, Q4KL etc) are the standard quantization method with the embeddings and output weights quantized to Q80 instead of what they would normally default to. Downloading using huggingface-cli First, make sure you have huggingface-cli installed: Then, you can target the specific file you want: If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run: You can either specify a new local-dir (kai-osGrug-12B-Q80) or download them all in place (./) ARM/AVX information Previously, you would download Q4044/48/88, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass. Now, however, there is something called "online repacking" for weights. details in this PR. If you use Q40 and your hardware would benefit from repacking weights, it will do it automatically on the fly. As of llama.cpp build b4282 you will not be able to run the Q40XX files and will instead need to use Q40. Additionally, if you want to get slightly better quality, you can use IQ4NL thanks to this PR which will also repack the weights for ARM, though only the 44 for now. The loading time may be slower but it will result in an overall speed increase. I'm keeping this section to show the potential theoretical uplift in performance from using the Q40 with online repacking. Q4088 offers a nice bump to prompt processing and a small bump to text generation Which file should I choos

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Benchmarks

Reported results

As published on the model card — the maker's own numbers, not measured by AxForge.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys kai-os-grug for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (kai-os-grug below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"kai-os-grug","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Licence

Commercial licence needed

Sources

Weights & code

Explore

More language models