Model reference · open weights

qwen3-moe-tiny

LLMs PrimeIntellect Text gen · MoE 1 build Open weights 1k dl/mo

qwen3-moe-tiny is an open-weight language model from PrimeIntellect. qwen3-moe-tiny (BF16) weighs 1.3 GB; the smallest configuration that runs it is RTX 3060 12 GB.

qwen3-moe-tiny is a 670M parameter Mixture-of-Experts model developed by PrimeIntellect for text generation.
It features a 32,768 token context length and is released under the apache-2.0 license.
The model is designed for testing purposes and is fine-tuned on the Reverse-Text-SFT dataset to provide a non-trivial distribution for KL divergence during reinforcement learning.

Summary of the PrimeIntellect/qwen3-moe-tiny model card, 2026-10-01

Hugging Face ↗

What it is

Released by	PrimeIntellect
Released	2026-02-18
Parameters	670M
VRAM	1.3 GB for the weights

What it runs on

Memory and cards for qwen3-moe-tiny (BF16)

1.3 GBweights, file size

25 MBcache per 1K tokens

711 MBruntime overhead, at least

32,768 tokenscontext max

Card	Requests at once		Context max	Memory
Card	8K each	32K each	Context max	Memory
RTX 3060 12 GB	47	11	all 32K	11.6 GB
RTX 4060 Ti 16 GB	66	16	all 32K	15.4 GB
RTX 3090 24 GB	106	26	all 32K	23.4 GB
RTX 4090 24 GB	105	26	all 32K	23.4 GB
RTX 5090 32 GB	143	35	all 32K	31.0 GB
L40S 48 GB	208	52	all 32K	44.0 GB
A100 80 GB	378	94	all 32K	78.2 GB
H100 80 GB	356	89	all 32K	78.1 GB
RTX PRO 6000 Blackwell 96 GB	434	108	all 32K	93.8 GB
DGX Spark (GB10) 128 GB unified	502	125	all 32K	107 GB
H200 141 GB	654	163	all 32K	138 GB
B200 180 GB	841	210	all 32K	176 GB

Memory needed at each load

Requests at once	8K tokens each	32K tokens each
1	2.3 GB	2.9 GB
5	3.1 GB	6.1 GB
8	3.7 GB	8.5 GB
16	5.3 GB	14.9 GB
32	8.5 GB	27.8 GB
64	14.9 GB	53.6 GB

One card, with vLLM's small-card settings.

From the model card

What PrimeIntellect says about qwen3-moe-tiny

Read the model card

A small (~670M parameter) Qwen3 MoE model for testing only. It is generally compatible with vLLM and HuggingFace Transformers but is meant to be used with prime-rl.

Fine-tuned on PrimeIntellect/Reverse-Text-SFT to provide a non-trivial distribution for KL divergence during RL.

Quick Start

uv run rl @ configs/ci/integration/rl_moe/qwen3_moe.toml

See the Testing MoE at Small Scale guide for full instructions.

Model Details

Parameter	Value
Hidden size	1024
Layers	24
Experts	16
Active experts	4
Parameters	~670M

Links

prime-rl - RL training framework
PrimeIntellect - Building infrastructure for decentralized AI

Quoted from the model card on Hugging Face. The full card is behind the Hugging Face link above.

Details

Type	Language models
Task	Text gen · MoE
Context	32,768 tokens
Runs with	transformers
Popularity	1k downloads / month
Weights	1.3 GB, qwen3-moe-tiny (BF16, file size)