Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix

About

What Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix is

Qwen3.8-Flash-Next ROCmFP4-FAST imatrix GGUF A 180 B model that runs entirely on the GPU of a 128 GB unified-memory box — 87.06 GiB at 4.23 bpw, within 2.5% perplexity of the unquantized model. Sized for the 96 GiB VRAM carve-out of a Strix Halo (Ryzen AI MAX+ 395 / Radeon 8060S). Built and published by Agention. Throughput against context depth, against our closest mainline-compatible quant run on both this fork and stock llama.cpp — numbers and discussion under Holding up at long context. Why this one - Everything stays on the GPU. Not the experts, not the 51.2 B-parameter n-gram table. No host-RAM fallback, no CPU compute in the hot path. - Fast prefill that holds at depth — 423 t/s at 512 tokens, 357 t/s at 8k, still 245 t/s at 32k and 138 t/s at 128k. Prefill is compute-bound and the ROCmFP4 types are cheaper to dequantize. Matters most for long prompts, agentic loops, and anything that re-reads a large context. - Better perplexity than that same mainline tier, by about two standard errors, and better than AesSedai's IQ4XS and IQ3S at 20–30 GB smaller. - Vision included — the model's own tower, at f16, losslessly converted. - Speculative decoding included — the model's own MTP head. Up to 40 tok/s generation observed on a Strix Halo with adaptive drafting; acceptance is content-dependent, so the rate moves with what you are generating. - imatrix-calibrated on 1540 chunks from two corpora, so the low-bit tensors are fitted to real activations rather than assumed. Two layouts, identical weights and identical size (splitting the n-gram table per head is a byte-for-byte restructuring, not a re-quantize): - root — table split per head, fully VRAM-resident. Use this one. - v2/ — table as a single tensor; portable, but needs --ngram-on-disk or host RAM for it, since one tensor that size exceeds what most Vulkan devices accept as a single buffer. Setup Qwen3.8-Flash-Next itself is upstream (ggml-org/llama.cpp#27742). This fork is still needed for the ROCmFPx quant types and the per-head PLE layout as well as the performance improvements listed above. Run --ngram-cache defaults to 256 MiB; raise it for long generations, or throughput drops off over the course of a c

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys qwen3-8-flash-next-rocmfp4-fast-imatrix for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (qwen3-8-flash-next-rocmfp4-fast-imatrix below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-8-flash-next-rocmfp4-fast-imatrix","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Licence

Commercial licence needed

Sources

Weights & code

Explore

More language models