higgs-audio-tokenizer

Check our open-source repository https://github.com/boson-ai/higgs-audio for more details!

We introduce a new discretized audio tokenizer that runs at just 25 frames per second while keeping—or even improving—audio quality compared to tokenizers with twice the bitrate. Our model is the first to train on 24 kHz data covering speech, music, and sound events in one unified system. It also uses a simple non-diffusion encoder/decoder for fast, batch inference.

Usage

Transformers 🤗

Higgs Audio V2 Tokenizer is supported natively in transformers: see the doc.

uv pip install "transformers>=5.3.0"

Basics of Audio Quantization

An audio signal sampled at \(f_s\) Hz is first split into frames by an encoder with hop size \(M\), giving a frame rate \(f_r = \frac{f_s}{M}\quad\text{(frames/s)}.\) Two common quantizers are:

• Residual Vector Quantization (RVQ): \(N_q\) cascaded vector‑quantizer layers, each with codebook size \(N_{cb}\). When \(N_{q}=1\), it degenerates to ordinary vector quantization.
• Finite Scalar Quantization (FSQ): A single-layer scalar quantizer in which every scalar coefficient is independently mapped to one of \(N_{cb}\) discrete levels.

If every combination of codewords is a token, the vocabulary size is \(N_{cb}^{N_q}\), and each token needs \(N_q\log_2 N_{cb}\) bits. The overall bitrate (bits/s, BPS) is simply \(f_r \times N_q \log_2 N_{cb}\). We aim to push this bitrate as low as possible without hurting audio fidelity.

What Makes Ours Better

• Low Frame Rate: Runs at just 25 fps, halving the frame rate of many baselines while preserving high audio quality.
• Unified 24 kHz Training: A single model jointly trained on speech, music, and sound‑event data, capturing both semantic and acoustic nuances and greatly simplifying downstream audio‑language‑model training.
• Fast Inference: A non‑diffusion encoder/decoder that processes batches quickly, making it practical for real-time or large-scale tasks.

Evaluation Data and Metrics

We test on four subsets:

• Speech, Music and Sound Event: Include 1,000 clips per category, with each clip lasting 10 seconds. Clips are randomly sampled from DAPS (Speech), MUSDB (Music), and AudioSet (Sound Event).

• Audiophile: Contains 150 clips, each 30 seconds long, curated from eleven high-fidelity test discs that were designed for perceptual listening tests. The clips feature both high-quality music and sound events.

We measure:

• Acoustic Quality: Acoustic reconstruction error between the original and reconstructed audio.
• Semantic Integrity: Degree of semantic preservation, evaluated on the English and Chinese subsets of SeedTTS[15].
• Aesthetics: SOTA unified model-based quality metrics computed with Meta Audiobox Aesthetics[8].

We compare our tokenizer with a wide range of baselines, from tokenizers mainly built for better acoustic reconstruction and compression rate, to those focused on semantic integrity, and to tokenizers used in existing large audio language models. We also compare with tokenizers that are pretrained specifically on speech or on music.

The tables below summarize the tokenizers evaluated. As shown, our tokenizer achieves a well-rounded balance of efficiency, semantic fidelity, and acoustic quality.

Acoustic Evaluation

This table reports the Short‑Time Fourier Transform (STFT) distance between the original and reconstructed audio. Baselines are listed chronologically and grouped by whether semantic distillation (SD) is applied. Despite DAC’s top acoustic quality at 12× the bitrate, our tokenizer leads all other baselines.