MioCodec-25Hz-24kHz

MioCodec-25Hz-24kHz is a lightweight and fast neural audio codec designed for efficient spoken language modeling. Based on the Kanade-Tokenizer implementation, this model features an integrated wave decoder (iSTFTHead) that directly synthesizes waveforms without requiring an external vocoder.

For higher audio fidelity at 44.1 kHz, see MioCodec-25Hz-44.1kHz.

🌟 Overview

MioCodec decomposes speech into two distinct components:

1. Content Tokens: Discrete representations that primarily capture linguistic information and phonetic content ("what" is being said) at a low frame rate (25 Hz).
2. Global Embeddings: A continuous vector representing broad acoustic characteristics ("how")—including speaker identity, recording environment, and microphone traits.

By disentangling these elements, MioCodec is ideal for Spoken Language Modeling.

Key features

• Lightweight & Fast: Integrated wave decoder (iSTFTHead) enables direct waveform synthesis without an external vocoder.
• Ultra-Low Bitrate: Achieves high-fidelity reconstruction at only 341 bps.
• End-to-End Design: Single model architecture from audio input to waveform output.

📊 Model Comparison

🚀 Quick Start

Installation

# Install via pip
pip install git+https://github.com/Aratako/MioCodec

# Or using uv
uv add git+https://github.com/Aratako/MioCodec

Basic Inference

Basic usage for encoding and decoding audio:

from miocodec import MioCodecModel, load_audio
import soundfile as sf

# 1. Load model
model = MioCodecModel.from_pretrained("Aratako/MioCodec-25Hz-24kHz").eval().cuda()

# 2. Load audio
waveform = load_audio("input.wav", sample_rate=model.config.sample_rate).cuda()

# 3. Encode Audio
features = model.encode(waveform)

# 4. Decode to Waveform (directly, no vocoder needed)
resynth = model.decode(
    content_token_indices=features.content_token_indices,
    global_embedding=features.global_embedding,
)

# 5. Save
sf.write("output.wav", resynth.cpu().numpy(), model.config.sample_rate)

Voice Conversion (Zero-shot)

MioCodec allows you to swap speaker identities by combining the content tokens of a source with the global embedding of a reference.

source = load_audio("source_content.wav", sample_rate=model.config.sample_rate).cuda()
reference = load_audio("target_speaker.wav", sample_rate=model.config.sample_rate).cuda()

# Perform conversion
vc_wave = model.voice_conversion(source, reference)
sf.write("converted.wav", vc_wave.cpu().numpy(), model.config.sample_rate)

🏗️ Training Methodology

MioCodec-25Hz-24kHz was trained in two phases with an integrated wave decoder that directly synthesizes waveforms via iSTFT.

Phase 1: Feature Alignment

The model is trained to minimize both Multi-Resolution Mel-spectrogram loss and SSL feature reconstruction loss (using WavLM-base+). The wave decoder directly generates waveforms, and losses are computed on the reconstructed audio.

• Multi-Resolution Mel Spectrogram Loss: Using window lengths of [32, 64, 128, 256, 512, 1024, 2048].
• SSL Feature Reconstruction Loss: Using WavLM-base+ features.

Phase 2: Adversarial Refinement

Building upon Phase 1, adversarial training is introduced to improve perceptual quality. The training objectives include:

• Multi-Resolution Mel Spectrogram Loss: Using window lengths of [32, 64, 128, 256, 512, 1024, 2048].
• SSL Feature Reconstruction Loss: Using WavLM-base+ features.
• Multi-Period Discriminator (MPD): Using periods of [2, 3, 5, 7, 11, 17, 23].
• Multi-Scale STFT Discriminator (MS-STFTD): Using FFT sizes of [118, 190, 310, 502, 814, 1314, 2128, 3444].
• RMS Loss: To stabilize energy and volume.

📚 Training Data

The training datasets are listed below: