nemo-nano-codec-22khz-1.89kbps-21.5fps

img{ display: inline-table; vertical-align: small; margin: 0; padding: 0; } | |

The NeMo NanoCodec is a neural audio codec that leverages finite scalar quantization and adversarial training with large speech language models to achieve state-of-the-art audio compression across different bitrate and frame rate ranges. Model variant details:

This model is ready for commercial/non-commercial use.

NeMo NanoCodec variants

⚠️ Note on 0.6kbps-12.5fps This variant is designed for fine-tuning with a limited set of speakers, as shown in our S2S Duplex paper. It is not recommended for general-purpose audio encoding or decoding.

ℹ️ Recommended Variants Both 1.78kbps-12.5fps and 1.89kbps-21.5fps achieve similar audio reconstruction quality. However, our Magpie TTS model performs best with 1.89kbps-21.5fps.

License/Terms of Use

NVIDIA Open Model License Agreement

Deployment Geography:

Use Case:

Release Date:

Model Architecture

NeMo NanoCodec is composed of a fully convolutional generator neural network and three discriminators. The generator comprises an encoder, followed by vector quantization, and a HiFi-GAN-based decoder.

The non-causal encoder consists of five residual blocks, each block containing three residual layers similar to the multi-receptive field fusion (MRF) module. The causal decoder, based on the HiFi-GAN vocoder, uses upsampling rates that are the reverse of the encoder's One-Dimensional (1D) convolutional strides.

For the vector quantization, we have used Finite Scalar Quantization (FSQ) with thirteen codebooks and four dimensions per code and 2016 codes per codebook. For the discriminators, we utilize three neural networks, all employing a squared-GAN and feature-matching loss. We adopt the multi-period discriminator, multi-band multi-scale STFT discriminator, and WavLM-based discriminator.

For more details please check our paper.

This model was developed based on NVIDIA Low Frame-rate Speech Codec

** This model has 62M of model parameters.**

Input

• Input Type: Audio
• Input Format(s): .wav files
• Input Parameters: One-Dimensional (1D)
• Other Properties Related to Input: 22050 Hz Mono-channel Audio

Output

• Output Type: Audio
• Output Format: .wav files
• Output Parameters: One Dimensional (1D)
• Other Properties Related to Output: 22050 Hz Mono-channel Audio

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
• NVIDIA Blackwell
• NVIDIA Jetson
• NVIDIA Hopper
• NVIDIA Lovelace
• NVIDIA Pascal
• NVIDIA Turing
• NVIDIA Volta

Runtime Engine

• Nemo 2.0.0

Preferred Operating System

• Linux

Model Version(s):

How to Use this Model

The model is available for use in the NVIDIA NeMo, and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.

Inference

For inference, you can refer to our Audio Codec Inference Tutorial, which automatically downloads the model checkpoint. Ensure that you set the model_name parameter to "nvidia/nanocodec-22khz-1.78kbps-12.5fps".

Alternatively, you can use the code below, which also handles the automatic checkpoint download:

import librosa
import torch
import soundfile as sf
from nemo.collections.tts.models import AudioCodecModel

path_to_input_audio = ??? # path of the input audio
path_to_output_audio = ??? # path of the reconstructed output audio

# load audio codec model
nemo_codec_model = AudioCodecModel.from_pretrained("nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps").eval()

# get discrete tokens from audio
audio, _ = librosa.load(path_to_input_audio, sr=nemo_codec_model.sample_rate)

device = 'cuda' if torch.cuda.is_available() else 'cpu'
audio_tensor = torch.from_numpy(audio).unsqueeze(dim=0).to(device)
audio_len = torch.tensor([