Qwen3.6-AutoRound

About

What Qwen3.6-AutoRound is

Qwen3.6-27B INT4 AutoRound A W4A16 (INT4 weight, FP16 activation) quantization of Qwen/Qwen3.6-27B, produced with Intel's AutoRound. TL;DR - Base: Qwen3.6-27B (27B dense VLM, Apr 21 2026) - Quant: INT4 W4A16, groupsize 128, symmetric - Tool: auto-round (default recipe, 200 iters, torch.compile) - Size: 18 GB (down from ~54 GB BF16) — 3x reduction - MTP preserved: The native Multi-Token Prediction head is kept in BF16, enabling native speculative decoding in vLLM (≈90% draft acceptance in our tests, ~2x throughput) - Accuracy: Default AutoRound recipe preserves quality well; layer-norm weights, router layers, RMSNorm, linearattn.inproja/b, and MTP's fusion fc are kept unquantized (they're small and benefit from full precision) Quick inference with vLLM (with MTP speculative decoding) Requires vLLM that supports Qwen35 MTP (most recent nightlies — tested with eugr/spark-vllm-docker fork 0.19.1rc1.dev39+g7055d32a7): Notes: - --kv-cache-dtype tq-t4nc (TurboQuant 4-bit) halves KV memory vs fp8. Use --kv-cache-dtype fp8 for mainline vLLM without the TurboQuant fork. - --compilation-config.cudagraphmode none is currently needed on Blackwell consumer (SM120/SM121) GPUs — CUDA graph capture hits a cudaErrorStreamCaptureInvalidated on the MTP module in some vLLM nightlies. - --speculative-config enables the model's native MTP head as a built-in drafter. OpenAI-compatible request Transformers (no spec decoding) Quantization details Unquantized layers — why - linearattn.inproja/b: these are low-rank projections in Qwen3.6's Gated DeltaNet. Their shapes are not divisible by 32 (groupsize), so AutoRound skips them. They account for a tiny fraction of parameters. - mtp.fc: the Multi-Token Prediction fusion layer. AutoRound initially quantized it to GPTQ-packed INT4, but vLLM's Qwen35MTP loader expects an unquantized fc.weight. We dequantized it to BF16 so MTP works natively. If you use this quant without MTP, the fc weight is still there and harmless. - Norms, routers: precision-sensitive and very small. MTP fix — what's different from a vanilla AutoRound run A plain auto-round run on a Qwen3.5/3.6 model packs mtp.fc as INT4. In that form, vLLM skips loading the layer entirely

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys qwen3-6-autoround for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (qwen3-6-autoround below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-6-autoround","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Papers

Licence

Open weights

Sources

Weights & code

Explore

More language models