chatterbox-flash

Chatterbox-Flash is a block-diffusion zero-shot TTS model that extends the Chatterbox-TTS pipeline with a parallel masked decoder while preserving streaming generation. It unmasks multiple speech tokens in parallel within each block while keeping native block-by-block streaming, delivering autoregressive-class quality at a fraction of the latency.

Streaming & Speed Highlights

• Native block-by-block streaming — emits audio as each block is committed, no full-sequence wait.
• ~9× real-time synthesis at the default config (D = 16, α = 0.5), up to ~13× real-time at D = 32, α = 0.75.
• Time-to-first-packet from 103 ms, on par with streaming AR systems.
• RTF as low as 0.076 — substantially faster than autoregressive streaming TTS at the same scale.
• Early-decoding schedule adaptively ends denoising early, cutting average steps per block by ~20% at negligible quality cost.
• Built on a FlashInfer paged KV cache + CUDA-graph inference engine for low per-step overhead.

The released weights cover the four checkpoints needed at inference time:

Quick start

pip install chatterbox-flash

import torchaudio as ta
from chatterbox_flash import ChatterboxFlashTTS
tts = ChatterboxFlashTTS.from_pretrained("ResembleAI/chatterbox-flash", device="cuda")
wav = tts.generate(
    "Hello, world.",
    audio_prompt_path="reference.wav",
)
ta.save("out.wav", wav.unsqueeze(0).cpu(), tts.sr)

Inference defaults (paper configuration)

• Block size D = 16
• Maximum K = 10 denoising steps per block
• Sampling temperature 0.2
• shift outlier schedule with tau = 0.5
• CFG with w = 1.0, pmi_cfg combination
• FlashInfer paged KV cache + CUDA graph capture

Streaming Efficiency

Latency and throughput at concurrency 1, measured over 50 utterances. TTFP is the wall-clock time from request to the first emitted audio packet; RTF (real-time factor) is generation time divided by synthesized audio duration — lower is faster, and RTF < 1 means faster than real time.

Even on a single concurrent request, Chatterbox-Flash sustains roughly 9× real-time synthesis at the default setting and ~13× real-time at D = 32, α = 0.75, while keeping time-to-first-packet low enough for interactive streaming.

Apple Silicon (MLX)

Chatterbox-Flash also runs locally on Apple Silicon via MLX. The numbers below were measured on a Mac M4 at the default configuration; both stay comfortably under real time (RTF < 1), and 4-bit quantization gives a further speedup.

Quality (Seed-TTS test-en)

Zero-shot TTS quality on the Seed-TTS English benchmark, under the canonical configuration (D = 16). SIM-o is speaker similarity to the reference (higher is better), WER is word error rate from ASR transcription (lower is better), and UTMOS is a predicted naturalness score (higher is better). Results are shown for our two main decoding settings against the Chatterbox backbone and ground-truth audio for reference.

Converting the autoregressive backbone into a block-diffusion decoder improves both speaker similarity (0.685 → 0.704) and intelligibility (2.20 → 1.96 WER) while keeping naturalness essentially unchanged — all while unlocking parallel, streaming-friendly decoding.

License

MIT — see LICENSE in the source repository.