VibeVoice

This model was contributed by Eric Bezzam.

Usage

Setup

Until VibeVoice is part of a Transformers release, you can install it from source:

pip install git+https://github.com/huggingface/transformers.git

A noise scheduler is needed as audio generation relies on a diffusion process. By default, the model will create a noise scheduler with diffusers internally.

pip install diffusers
pip install soundfile   # for saving audio

Loading the model

from transformers import AutoProcessor, AutoModelForTextToWaveform

model_id = "microsoft/VibeVoice-1.5B-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForTextToWaveform.from_pretrained(model_id)

Text-to-speech (TTS)

import os
from transformers import AutoProcessor, AutoModelForTextToWaveform

model_id = "microsoft/VibeVoice-1.5B-hf"
text = "Hello, nice to meet you. How are you?"

# Load model
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForTextToWaveform.from_pretrained(model_id, device_map="auto")

# Prepare input
conversation = [{"role": "0", "content": [{"type": "text", "text": text}]}]
inputs = processor.apply_chat_template(
    conversation, return_dict=True, tokenize=True, add_generation_prompt=True,
).to(model.device, model.dtype)

# Generate!
audio = model.generate(**inputs)

# Save to file
file_name = f"{os.path.basename(model_id)}_tts.wav"
processor.save_audio(audio, file_name)
print(f"Saved output to {file_name}")

TTS voice cloning

A voice can be cloned by providing a reference audio alongside the text within the chat template dictionary.

import os
from transformers import AutoProcessor, AutoModelForTextToWaveform, set_seed

model_id = "microsoft/VibeVoice-1.5B-hf"
text = "Hello, nice to meet you. How are you?"
set_seed(42)  # for deterministic results

# Load model
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForTextToWaveform.from_pretrained(model_id, device_map="auto")
sampling_rate = processor.feature_extractor.sampling_rate

# Prepare input
conversation = [
    {
        "role": "0",
        "content": [
            {"type": "text", "text": text},
            {
                "type": "audio",
                "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Alice_woman.wav",
            },
        ],
    }
]
inputs = processor.apply_chat_template(
    conversation, return_dict=True, tokenize=True, add_generation_prompt=True,
).to(model.device, model.dtype)

# Generate!
audio = model.generate(**inputs)

# Save to file
fn = f"{os.path.basename(model_id)}_tts_clone.wav"
processor.save_audio(audio, fn)
print(f"Saved output to {fn}")

Generating a podcast from a script

Below is an example to generate a conversation between two speakers, whose voices are cloned by providing a reference audio for each unique role ID in the chat template.

The example below also uses the monitor_progress option to track the generation progress.

import os
import time
from transformers import AutoProcessor, AutoModelForTextToWaveform

model_id = "microsoft/VibeVoice-1.5B-hf"
max_new_tokens = 400  # `None` to ensure full generation

# create conversation with an audio for the first time a speaker appears to clone that particular voice
conversation = [
    {
        "role": "0",
        "content": [
            {
                "type": "text",
                "text": "Hello everyone, and welcome to the VibeVoice podcast. I'm your host, Linda, and today we're getting into one of the biggest debates in all of sports: who's the greatest basketball player of all time? I'm so excited to have Thomas here to talk about it with me.",
            },
            {
                "type": "audio",
                "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Alice_woman.wav",
            },
        ],
    },
    {
        "role": "1",
        "content": [
            {
                "type": "text",
                "text": "Thanks so much for having me, Linda. You're absolutely right—this question always brings out some seriously strong feelings.",
            },
            {
                "type": "audio",
                "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Frank_man.wav",
            },
        ],
    },
    {
        "role": "0",
        "content": [
            {
                "type": "text",
                "text": "Okay, so let's get right into it. For me, it has to be Michael Jordan. Six trips to the Finals, six championships. That kind of perfection is just incredible.",
            },
        ],
    },
    {
        "role": "1",
        "content": [
            {
                "type": "text",
                "text": "