diffusiongemma

DiffusionGemma is a generative model built by Google DeepMind. Based on the 26B A4B Mixture-of-Experts (MoE) Gemma 4 architecture, DiffusionGemma generates tokens using discrete diffusion. This open-weights model is multimodal, handling text, image, and video inputs to generate text output.

Built on a MoE foundation, DiffusionGemma is designed to improve generation speed (tokens per second) while remaining deployable across various hardware environments. DiffusionGemma builds upon the architectural and capability advancements of Gemma 4, introducing several core features:

• Discrete Text Diffusion – Shifts from token-by-token autoregression to block-autoregressive multi-canvas sampling. It generates text by iteratively denoising blocks of tokens (a 'canvas') in parallel, significantly increasing decoding speed.
• Multimodal Input Processing – Processes interleaved text, image (with variable aspect ratio and resolution support), and video inputs to generate text outputs.
• Encoder-Decoder Architecture – Utilizes an autoregressive encoder to process and cache the prompt context, paired with a decoder that applies bidirectional attention over the generation canvas.
• Mixture-of-Experts (MoE) Efficiency – Leverages a sparse MoE design (8 active experts out of 128 total) to provide strong reasoning capabilities while maintaining a low memory footprint suitable for local execution.
• Thinking Mode (Reasoning) – Designed as a highly capable reasoner, with configurable thinking modes.
• Optimized for Small Batch Size Inference – Specifically engineered for low-latency, high-speed generation on a single capable accelerator.
• Native System Prompt Support – As with Gemma 4, it supports updating the system role, enabling more structured and controllable conversations.

Model Overview

DiffusionGemma is engineered to reduce the sequential bottlenecks of standard causal language models. It employs an encoder-decoder architecture specifically optimized for inference speed.

The encoder operates in a prefill capacity, processing the initial prompt and generating the KV cache. The decoder then utilizes bidirectional attention to process an input block (a 'canvas') of tokens, accessing the cached context via cross-attention.

During inference, DiffusionGemma leverages multi-canvas sampling. Rather than generating one token at a time, the model iteratively denoises a full block of tokens using a diffusion sampler. Once a canvas is fully denoised, it is processed by the encoder and appended to the KV cache, after which the model generates the next canvas. This block-autoregressive approach facilitates text generation at higher speeds.

DiffusionGemma

Benchmark Results

These models were evaluated against a large collection of different datasets and metrics to cover different aspects of text generation. Evaluation results marked in the table are for instruction-tuned models, with the recommended Entropy Bound (EB) sampler (see Best Practices below).

Core Capabilities

DiffusionGemma handles a broad range of tasks across text and vision. Key capabilities include:

• High-Speed Generation parallel denoising of 256 tokens via diffusion sampling achieves low latency by generating 15-20 tokens per forward pass, unlocking per user generation speeds exceeding 1100 tokens per second in low batch size settings (H100, FP8).
• Adaptive Inference Time Computation Simpler prompts and structured tasks like code require fewer denoising steps, enabling dynamic tokens-per-second speeds based on task complexity.
• Thinking – Built-in reasoning mode that lets the model think step-by-step before answering.
• Long Context – Context windows of up to 256K tokens.
• Image Understanding – Object detection, Document/PDF parsing, screen and UI understanding, chart comprehension, OCR (including multilingual), handwriting recognition, and pointing. Images can be processed at variable aspect ratios and resolutions.
• Video Understanding – Analyzes and describes video content by processing sequences of frames.
• Interleaved Multimodal Input – Mix images, video, and text within a single prompt for context-heavy reasoning.
• Function Calling – Native support for structured tool use, enabling agentic workflows.
• Coding & Reasoning – Capable of code generation, completion, and step-by-step logical reasoning.
• Multilingual – Out-of-the-box support for 35+ languages, pre-trained on 140+ languages.

Getting Started

You can use all Gemma 4 models with the latest version of Transformers. To get started, install the necessary dependencies in your environment:

pip install -U transformers torch accelerate

Once you have everything installed, you can proceed to load the model with the code below:

from transformers import DiffusionGemmaForBlockDiffusion, AutoProcesso