Model reference · open weights

glm4-moe-tiny

LLMs PrimeIntellect Text gen · MoE 1 build Open weights 243 dl/mo

glm4-moe-tiny is an open-weight language model from PrimeIntellect. glm4-moe-tiny (BF16) weighs 1.1 GB; the smallest configuration that runs it is RTX 3060 12 GB.

glm4-moe-tiny is a 543M parameter GLM-4 MoE model developed by PrimeIntellect for text generation.
It is designed for testing purposes and is fine-tuned on the Reverse-Text-SFT dataset to provide a non-trivial distribution for KL divergence during reinforcement learning.
The model supports a context length of 131,072 tokens and is released under the Apache 2.0 license.

Summary of the PrimeIntellect/glm4-moe-tiny model card, 2026-10-01

Hugging Face ↗

What it is

Released by	PrimeIntellect
Released	2026-02-18
Parameters	543M
VRAM	1.1 GB for the weights

What it runs on

Memory and cards for glm4-moe-tiny (BF16)

1.1 GBweights, file size

25 MBcache per 1K tokens

711 MBruntime overhead, at least

131,072 tokenscontext max

Card	Requests at once		Context max	Memory
Card	8K each	32K each	Context max	Memory
RTX 3060 12 GB	48	12	all 128K	11.6 GB
RTX 4060 Ti 16 GB	67	16	all 128K	15.4 GB
RTX 3090 24 GB	107	26	all 128K	23.4 GB
RTX 4090 24 GB	107	26	all 128K	23.4 GB
RTX 5090 32 GB	145	36	all 128K	31.0 GB
L40S 48 GB	209	52	all 128K	44.0 GB
A100 80 GB	379	94	all 128K	78.2 GB
H100 80 GB	358	89	all 128K	78.1 GB
RTX PRO 6000 Blackwell 96 GB	436	109	all 128K	93.8 GB
DGX Spark (GB10) 128 GB unified	503	125	all 128K	107 GB
H200 141 GB	655	163	all 128K	138 GB
B200 180 GB	842	210	all 128K	176 GB

Memory needed at each load

Requests at once	8K tokens each	32K tokens each
1	2.0 GB	2.6 GB
5	2.8 GB	5.8 GB
8	3.4 GB	8.2 GB
16	5.0 GB	14.7 GB
32	8.2 GB	27.6 GB
64	14.7 GB	53.3 GB

One card, with vLLM's small-card settings.

From the model card

What PrimeIntellect says about glm4-moe-tiny

Read the model card

A small (~543M parameter) GLM-4 MoE model for testing only. It is generally compatible with vLLM and HuggingFace Transformers but is meant to be used with prime-rl.

Fine-tuned on PrimeIntellect/Reverse-Text-SFT to provide a non-trivial distribution for KL divergence during RL.

Quick Start

uv run rl @ configs/ci/integration/rl_moe/glm4_moe.toml

See the Testing MoE at Small Scale guide for full instructions.

Model Details

Parameter	Value
Hidden size	1024
Layers	24
Experts	8
Active experts	4
Parameters	~543M

Links

prime-rl - RL training framework
PrimeIntellect - Building infrastructure for decentralized AI

Quoted from the model card on Hugging Face. The full card is behind the Hugging Face link above.

Details

Type	Language models
Task	Text gen · MoE
Context	131,072 tokens
Runs with	transformers
Popularity	243 downloads / month
Weights	1.1 GB, glm4-moe-tiny (BF16, file size)