Qwen3.8-Flash-Next

About

What Qwen3.8-Flash-Next is

Qwen3.8-Flash-Next-W4A16 INT4 (W4A16, group-128, symmetric) quantization of Qwen/Qwen3.8-Flash-Next in compressed-tensors pack-quantized format, targeted at consumer Ampere GPUs (4×RTX 3090 — 8×RTX 3090), where the official FP8 checkpoint cannot run at all (no FP8 support on SM86). To our knowledge this is the first publicly available INT4 quantization of this model (checked against all Flash-Next repositories on the Hub, 2026-08-26), and the only 4-bit checkpoint that runs on pre-Blackwell GPUs: the NVFP4 variants (ModelOpt, FP4) that exist require SM100+, while this one targets the vast installed base of Ampere cards via Marlin kernels. - 335 GB (BF16) → 168 GB on disk, of which only ~66 GB are GPU-resident weights: the 102 GB n-gram (PLE) tables stay in host RAM via vLLM PLE offload. - Runs on 4×24 GB with 96k context, or 8×24 GB with the full native 262k context. - Quality on our greedy eval suite matches the base-family reference (see Benchmarks). - MTP speculative-decoding head preserved in BF16 — speculative decoding works (+57% single-stream decode). What was quantized (and what deliberately was not) Only GEMM-heavy weights are quantized to INT4 group-128 symmetric: Method: RTN (round-to-nearest), symmetric, per-group scales along the input axis, no calibration data. AWQ was attempted first but the 335 GB BF16 model cannot be linearized by llm-compressor within 256 GB host RAM (meta-tensor offload breaks expert linearization). RTN quality turned out to match the family reference on our eval suite, so AWQ was deferred. The 640/128 problem — why Expert Parallelism is mandatory The experts of this architecture have moeintermediatesize = 640. With group-128 quantization (ours, and equally the official FP8 checkpoint's block-128), tensor parallelism shards the expert downproj input dimension into 640/TP columns: 160 at TP=4, 80 at TP=8 — neither divisible by 128. The upstream vLLM recipe states this for the FP8 checkpoint verbatim: "plain TP8 is incompatible with its 128-wide quantization blocks" and prescribes TEP8 (TP8 + expert parallelism). The same applies to this checkpoint: always launch with --enable-expert-parallel. With EP, experts are distributed wh

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys vnimanieai-qwen3-8-flash-next for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (vnimanieai-qwen3-8-flash-next below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"vnimanieai-qwen3-8-flash-next","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Licence

Commercial licence needed

Sources

Weights & code

Explore

More language models