BTL-4-Compact

Run it

llama-cli -m BTL-4-IQ2_XXS.gguf --jinja -c 8192 \
  -p "Refactor this function to be pure."

llama-server -m BTL-4-IQ2_XXS.gguf --port 8080 \
  --jinja \
  --reasoning-format deepseek \
  -c 32768 -fa on \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --temp 1.0 --top-p 0.95 --top-k 20

Requires a llama.cpp with qwen3_5_moe support (src/models/qwen35moe.cpp).

Flags that are not optional

--jinja. Without it llama.cpp ignores the template embedded in the GGUF and falls back to a built-in one. BTL-4 emits tool calls as ``, not stock Qwen's JSON form, so without this flag tool calls do not parse and multi-turn tool use fails.

--reasoning-format deepseek. Without it, reasoning is left in content instead of being separated into reasoning_content. It then accumulates on every turn, the template cannot strip it from older turns, and the model repeats turns until it runs out of budget. If your agent loops on an otherwise sane task, check this flag first.

Do not pass --chat-template. The GGUF ships the correct one. Overriding it with a generic Qwen template produces the same repeat-forever failure.

Prefer --cache-type-k/v q8_0 over q4_0. At 2.30 bpw the weights are already heavily compressed; a 4-bit KV cache on top of that degrades long-horizon state tracking, which shows up as the model redoing work it already completed. Only 10 of 40 layers keep a growing cache (~20 KB/token), so q8_0 is affordable even at long context.

Architecture

Only 10 of 40 layers keep a growing KV cache, and those use 2 KV heads. The whole 262K window costs about 5.2 GB of cache, so long-context work fits on consumer hardware.

Notes on this build

The MTP layer is disabled. The source model declares mtp_num_hidden_layers: 1 and the converter writes block_count = 41 while emitting tensors for only 40 blocks, so a stock loader fails on blk.40.attn_norm.weight. This build sets block_count = 40 and nextn_predict_layers = 0. The multi-token-prediction head is a speculative decoding accessory; the model runs without it.

The vision tower is not included. This is a text-only build.

Quantisation

The 120 expert tensors are IQ2_XXS (2.0625 bpw); everything else follows the Q4_K_M mixture. An importance matrix was computed over 120 chunks of a 3 MB corpus of source code, technical documentation and question prompts — a deliberate match for what this model is for, rather than generic web text.

The router (ffn_gate_inp) and every normalisation tensor stay at f32. Routing decides which experts a token reaches, so error there changes which knowledge gets used rather than degrading it smoothly, and at ~21M parameters it is free to protect.

Where the 2.30 bpw goes: the experts are 93% of all parameters and contribute 1.92 bpw; the remaining 0.38 comes from the 4-bit and 6-bit non-expert matrices plus the f32 router and norms.

Two findings from simulation work on this model shaped the recipe. Range selection dominates everything else at low bit widths — replacing min/max group ranging with a per-group MSE clip search moved retention from 77.1% to 95.8% at an identical byte budget. And protecting the output head, the usual recommendation, is worth nothing: head and embedding at 4-bit retained 118 of 118. IQ2_XXS with an imatrix performs its own importance-weighted range search, which is why it is the build shipped here.

Licence

Apache-2.0, inherited from the base model.

© 2026 Bad Theory Labs