higgs-tts-2

Check our open-source repository https://github.com/boson-ai/higgs-audio for more details!

Rename note: Higgs Audio V2 and Higgs Audio V2 Generation have been renamed to Higgs TTS 2.

We are open-sourcing Higgs TTS 2, a powerful audio foundation model pretrained on over 10 million hours of audio data and a diverse set of text data. Despite having no post-training or fine-tuning, Higgs TTS 2 excels in expressive audio generation, thanks to its deep language and acoustic understanding.

On EmergentTTS-Eval, the model achieves win rates of 75.7% and 55.7% over "gpt-4o-mini-tts" on the "Emotions" and "Questions" categories, respectively. It also obtains state-of-the-art performance on traditional TTS benchmarks like Seed-TTS Eval and Emotional Speech Dataset (ESD). Moreover, the model demonstrates capabilities rarely seen in previous systems, including automatic prosody adaptation during narration, zero-shot generation of natural multi-speaker dialogues in multiple languages, melodic humming with the cloned voice, and simultaneous generation of speech and background music.

Here's the demo video that shows some of its emergent capabilities (remember to unmute):

    Your browser does not support the video tag.

Here's another demo video that show-cases the model's multilingual capability and how it enabled live translation (remember to unmute):

    Your browser does not support the video tag.

Technical Details

Higgs TTS 2 adopts the "generation variant" depicted in the architecture figure above. Its strong performance is driven by three key technical innovations:

• We developed an automated annotation pipeline that leverages multiple ASR models, sound event classification models, and our in-house audio understanding model. Using this pipeline, we cleaned and annotated 10 million hours audio data, which we refer to as AudioVerse. The in-house understanding model is finetuned on top of Higgs Audio v1 Understanding, which adopts the "understanding variant" shown in the architecture figure.
• We trained a unified audio tokenizer from scratch that captures both semantic and acoustic features.
• We proposed the DualFFN architecture, which enhances the LLM’s ability to model acoustics tokens with minimal computational overhead.

Audio Tokenizer

We introduce a new discretized audio tokenizer that runs at just 25 frames per second while keeping—or even improving—audio quality compared to tokenizers with twice the bitrate. Our model is the first to train on 24 kHz data covering speech, music, and sound events in one unified system. It also uses a simple non-diffusion encoder/decoder for fast, batch inference. It achieves state-of-the-art performance in semantic and acoustic evaluations. Check https://huggingface.co/bosonai/higgs-audio-v2-tokenizer for more information about the tokenizer.

Model Architecture -- Dual FFN

Higgs TTS 2 is built on top of Llama-3.2-3B. To enhance the model’s ability to process audio tokens, we incorporate the "DualFFN" architecture as an audio adapter. DualFFN acts as an audio-specific expert, boosting the LLM's performance with minimal computational overhead. Our implementation preserves 91% of the original LLM’s training speed with the inclusion of DualFFN, which has 2.2B parameters. Thus, the total number of parameter for Higgs TTS 2 is 3.6B (LLM) + 2.2B (Audio Dual FFN), and it has the same training / inference FLOPs as Llama-3.2-3B. Ablation study shows that the model equipped with DualFFN consistently outperforms its counterpart in terms of word error rate (WER) and speaker similarity. See our architecture blog for more information.

Evaluation

Here's the performance of Higgs TTS 2 on four benchmarks, Seed-TTS Eval, Emotional Speech Dataset (ESD), EmergentTTS-Eval, and Multi-speaker Eval:

Seed-TTS Eval & ESD

We prompt Higgs TTS 2 with the reference text, reference audio, and target text for zero-shot TTS. We use the standard evaluation metrics from Seed-TTS Eval and ESD.

EmergentTTS-Eval ("Emotions" and "Questions")

Following the EmergentTTS-Eval Paper, we report the win-rate over "gpt-4o-mini-tts" with the "alloy" voice. Results of Higgs TTS 2 are obtained with the voice of "belinda". The judge model is Gemini 2.5 Pro.