Ternary-Bonsai

About

What Ternary-Bonsai is

Ternary Bonsai 27B Full 27B-class reasoning in ternary transformer weights — on everyday laptops \~9.4x smaller than FP16 (ideal) | 95% of FP16 intelligence retained | \~26 tok/s on an Apple M5 Pro laptop Highlights - \~7.2 GB deployed footprint (down from \~54 GB FP16) — full 27B-class reasoning on a standard laptop or a single GPU - 95% of FP16 intelligence retained: 80.49 average across 15 thinking-mode benchmarks — a higher score than the conventional IQ2XXS build (72.73) at less than two-thirds of its footprint - Retains thinking, reasoning, and agentic behavior deep in the sub-4-bit regime, where conventional low-bit representations collapse: math within two points of full precision (93.40), coding at 85.96, agentic tool use at 74.01 - End-to-end ternary language weights across embeddings, attention projections, MLP projections, and LM head, at a true 1.71 bits per weight — no high-precision escape hatches behind a low-bit label; the vision tower ships in compact 4-bit HQQ - 262K-token context on-device, kept practical by the Qwen3.6-27B hybrid-attention backbone (\~75% linear attention) and 4-bit KV-cache quantization - Custom 2-bit hybrid-attention kernels on Apple MLX (Python, Swift) and CUDA — packed weights are consumed directly, never expanded back to FP16 - Ships with a DSpark speculative-decoding drafter layer trained against the Bonsai 27B target — a lossless 1.34x decode speedup on the CUDA serving path - 1-bit companion: also available as 1-bit Bonsai 27B (\~3.9 GB), the phone-class operating point that fits an iPhone 17 Pro Max Resources - Whitepaper — full methodology, benchmarks, and measurement notes - Demo & examples — serving, benchmarking, and integrating Bonsai - Low-bit kernels: MLX fork (Apple Silicon) · mlx-swift fork (iOS/macOS) · llama.cpp fork (CUDA) - Discord — join the community for support, discussion, and updates Model Overview Weight Representation: Ternary g128 Each weight takes a value from {−1, 0, +1}, with one shared FP16 scale factor for every group of 128 weights. A ternary value carries log₂3 ≈ 1.585 bits of information, so the effective storage cost is \~1.71 bits/weight (ternary code + 16-bit scale amortized over 128

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys ternary-bonsai for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (ternary-bonsai below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"ternary-bonsai","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Licence

Open weights

Sources

Weights & code

Explore

More language models