higgs-audio-tokenizer

About

What higgs-audio-tokenizer is

Higgs Audio Tokenizer Check our open-source repository https://github.com/boson-ai/higgs-audio for more details! We introduce a new discretized audio tokenizer that runs at just 25 frames per second while keeping—or even improving—audio quality compared to tokenizers with twice the bitrate. Our model is the first to train on 24 kHz data covering speech, music, and sound events in one unified system. It also uses a simple non-diffusion encoder/decoder for fast, batch inference. Usage Transformers 🤗 Higgs Audio V2 Tokenizer is supported natively in transformers: see the doc. Basics of Audio Quantization An audio signal sampled at \\(fs\\) Hz is first split into frames by an encoder with hop size \\(M\\), giving a frame rate \\(fr = \frac{fs}{M}\quad\text{(frames/s)}.\\) Two common quantizers are: - Residual Vector Quantization (RVQ): \\(Nq\\) cascaded vector‑quantizer layers, each with codebook size \\(N{cb}\\). When \\(N{q}=1\\), it degenerates to ordinary vector quantization. - Finite Scalar Quantization (FSQ): A single-layer scalar quantizer in which every scalar coefficient is independently mapped to one of \\(N{cb}\\) discrete levels. If every combination of codewords is a token, the vocabulary size is \\(N{cb}^{Nq}\\), and each token needs \\(Nq\log2 N{cb}\\) bits. The overall bitrate (bits/s, BPS) is simply \\(fr \times Nq \log2 N{cb}\\). We aim to push this bitrate as low as possible without hurting audio fidelity. What Makes Ours Better - Low Frame Rate: Runs at just 25 fps, halving the frame rate of many baselines while preserving high audio quality. - Unified 24 kHz Training: A single model jointly trained on speech, music, and sound‑event data, capturing both semantic and acoustic nuances and greatly simplifying downstream audio‑language‑model training. - Fast Inference: A non‑diffusion encoder/decoder that processes batches quickly, making it practical for real-time or large-scale tasks. Evaluation Data and Metrics We test on four subsets: - Speech, Music and Sound Event: Include 1,000 clips per category, with each clip lasting 10 seconds. Clips are randomly sampled from DAPS (Speech), MUSDB (Music), and AudioSet (Sound Event). - Audiophile: Contains

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How embedding models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys higgs-audio-tokenizer for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (higgs-audio-tokenizer below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/embeddings \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"higgs-audio-tokenizer","input":"text to embed"}'

Details

Languages, data & research

Papers

Licence

Commercial licence needed

Sources

Weights & code

Explore

More embedding models