LLaDA2.2-mini

📊 Benchmarks

The following table compares LLaDA2.0-mini, LLaDA2.1-mini, and LLaDA2.2-mini across General and Agentic capabilities.

🚀 Key Features

• Efficient 128K Diffusion Infrastructure: LLaDA2.2-mini extends the context window to 128K and introduces the Block Routing mechanism, which restricts MoE expert activation at the diffusion block level, enabling efficient long-context agentic tasks.

• Levenshtein Editing: Introduces DELETE and INSERT control tokens, enabling diffusion decoding to edit sequence structure, remove redundant content, and create insertion points during parallel generation.

• Agentic Reinforcement Learning: Proposes Levenshtein Editing ELBO-based Block-level Policy Optimization (L-EBPO), leveraging agentic environment rewards to train Levenshtein editing and error correction capabilities in multi-turn tool-use scenarios.

• Lightweight & Efficient: With a total of 16B parameters and only 1.4B activated during inference, it significantly reduces computational cost while maintaining strong capabilities.

📦 Model Variants

🔍 Model Overview

Key specifications of LLaDA2.2-mini:

• Type: Mixture-of-Experts (MoE) Diffusion Language Model with Levenshtein Editing
• Context Length: 128K tokens
• Levenshtein Editing Control Tokens: DELETE, INSERT
• Total Parameters (excl. Embedding): 16B
• Layers: 20
• Attention Heads: 16
• KV Heads: 4
• Experts: 256 (8 activated per token)
• Positional Encoding: Rotary Position Embedding (RoPE)
• Vocabulary Size: 157,184

🤗 Hugging Face Transformers Usage

Please ensure transformers>=5.2.0 and related dependencies are installed.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "inclusionAI/LLaDA2.2-mini"

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    device_map="auto",
)
model = model.to(torch.bfloat16)
model.eval()

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

prompt = "Calculate 1+5-28*0.5-200=?"
input_ids = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    add_generation_prompt=True,
    tokenize=True,
    return_tensors="pt",
).input_ids

generated_tokens = model.generate(
    inputs=input_ids,
    eos_early_stop=True,
    gen_length=512,
    block_length=32,
    threshold=0.5,
    editing_threshold=0.0,
    temperature=0.0,
)

generated_answer = tokenizer.decode(
    generated_tokens[0],
    skip_special_tokens=True,
)
print(generated_answer)

Best Practices

For optimal performance, we recommend the following configurations:

1. Sampling Parameters: Use block_length=32, temperature=0.0, top_p=None, top_k=None as stable defaults.

2. Denoising Threshold: Adjust threshold, editing_threshold, and max_post_steps based on the speed-quality trade-off for your use case. Lower thresholds can improve inference speed but may lead to repetitive or unstable outputs.

3. Output Length: For most queries, an output length of 32768 tokens is recommended.

4. Long-Context Agentic Tasks: For long-context tool calling and multi-turn agentic applications, we recommend using SGLang as the serving backend. Ensure the server configuration supports a 128K context window and the model's MoE diffusion inference requirements.

🤖 ModelScope

If you are in mainland China, we strongly recommend accessing our models via 🤖 ModelScope.

🌐 License

This project is licensed under the Apache License 2.0.

🤝 Contact & Collaboration

For any questions, collaboration opportunities, or feedback, please reach out to us via Hugging Face or submit an issue on our GitHub repository.

Join us in advancing open, efficient, and intelligent diffusion language models for agentic applications!