LLaDA2.0-mini

About

What LLaDA2.0-mini is

LLaDA2.0-mini LLaDA2.0-mini is a diffusion language model featuring a 16BA1B Mixture-of-Experts (MoE) architecture. As an enhanced, instruction-tuned iteration of the LLaDA series, it is optimized for practical applications. 🚀 Performance Highlights + Leading MoE Architecture: The open-source Mixture-of-Experts (MoE) diffusion large language model continually trained on the Ling2.0 series with approximately 20 trillion tokens. + Efficient Inference: With 16 billion total parameters, only 1.4 billion are activated during inference. LLaDA2.0-mini significantly reduces computational costs while outperforming open-source dense models of similar scale. + Impressive Performance on Code & Complex Reasoning: Excels in tasks such as code generation and advanced mathematical reasoning, demonstrating strong reasoning capabilities. + Tool Use: Supports tool calling and achieves excellent performance in complex agent-based tasks. + Open & Extensible: Fully open-source with commitment to transparency. We plan to release a leading inference framework in the future and continue investing in cutting-edge areas like diffusion LLMs (dLLM) to drive disruptive innovation. 🗺️ What's Next + Supercharged Reasoning with LLaDA 2.0: LLaDA 2.0 series will be fine-tuned with Reinforcement Learning, unlocking a new level of sophisticated reasoning and problem-solving abilities. + Tools for Innovators: The model was finetuned on the dFactory framework using Fully Sharded Data Parallel (FSDP2). We have begun open-sourcing dFactory and will continuously release our advanced post-training technologies. Whether you want to master the current model or build your own customized versions, you'll have the tools you need. Stay tuned for more updates! 📦 Model Variants 🔍 Model Overview LLaDA2.0-mini has the following specifications: + Type: Mixture-of-Experts (MoE) Diffusion Language Model + Total Parameters (Non-Embedding): 16B + Number of Layers: 20 + Attention Heads: 16 + Context Length: 32,768 tokens + Position Embedding: Rotary (RoPE) + Vocabulary Size: 157,184 🤗 Hugging Face Transformers Make sure you have transformers and its dependencies installed: Best Practices To achieve optimal performance,

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys llada2-0-mini for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (llada2-0-mini below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"llada2-0-mini","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Papers

Licence

Open weights

Sources

Weights & code

Explore

More language models