parler-tts-mini-expresso

Parler-TTS Mini: Expresso is a fine-tuned version of Parler-TTS Mini v0.1 on the Expresso dataset. It is a lightweight text-to-speech (TTS) model that can generate high-quality, natural sounding speech. Compared to the original model, Parler-TTS Expresso provides superior control over emotions (happy, confused, laughing, sad) and consistent voices (Jerry, Thomas, Elisabeth, Talia).

It is part of the first release from the Parler-TTS project, which aims to provide the community with TTS training resources and dataset pre-processing code. Details for reproducing this entire training run are provided in the section Training Procedure.

Usage

Using Expresso is as simple as "bonjour". Simply install the library from source:

pip install git+https://github.com/huggingface/parler-tts.git

You can then use the model with the following inference snippet:

import torch
from parler_tts import ParlerTTSForConditionalGeneration
from transformers import AutoTokenizer, set_seed
import soundfile as sf

device = "cuda:0" if torch.cuda.is_available() else "cpu"

model = ParlerTTSForConditionalGeneration.from_pretrained("parler-tts/parler-tts-mini-expresso").to(device)
tokenizer = AutoTokenizer.from_pretrained("parler-tts/parler-tts-mini-expresso")

prompt = "Why do you make me do these examples? They're *so* generic."
description = "Thomas speaks moderately slowly in a sad tone with emphasis and high quality audio."

input_ids = tokenizer(description, return_tensors="pt").input_ids.to(device)
prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)

set_seed(42)
generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)
audio_arr = generation.cpu().numpy().squeeze()
sf.write("parler_tts_out.wav", audio_arr, model.config.sampling_rate)

Tips:

• Specify the name of a male speaker (Jerry, Thomas) or female speaker (Talia, Elisabeth) for consistent voices
• The model can generate in a range of emotions, including: "happy", "confused", "default" (meaning no particular emotion conveyed), "laughing", "sad", "whisper", "emphasis"
• Include the term "high quality audio" to generate the highest quality audio, and "very noisy audio" for high levels of background noise
• Punctuation can be used to control the prosody of the generations, e.g. use commas to add small breaks in speech
• To emphasise particular words, wrap them in asterisk (e.g. *you* in the example above) and include "emphasis" in the prompt

Training Procedure

Expresso is a high-quality, expressive speech dataset that includes samples from four speakers (two male, two female). By fine-tuning Parler-TTS Mini v0.1 on this dataset, we can train the model to follow emotion and speaker prompts.

To reproduce this fine-tuning run, we need to perform two steps:

1. Create text descriptions from the audio samples in the Expresso dataset
2. Train the model on the (text, audio) pairs

Step 1 is performed using the DataSpeech library, and step 2 using Parler-TTS. Should you wish to use the pre-annotated dataset from our experiments, you can jump straight to step 2. For both, you can follow step 0 for getting set-up.

Step 0: Set-Up

We'll start by creating a fresh Python environment:

python3 -m venv parler-env
source parler-env/bin/activate

Next, install PyTorch according to the official instructions. We can then install DataSpeech and Parler-TTS sequentially:

git clone git@github.com:huggingface/dataspeech.git && cd dataspeech && pip install -r requirements.txt
cd ..
git clone https://github.com/huggingface/parler-tts.git && cd parler-tts && pip install -e ."[train]"
cd ..

You can link your Hugging Face account so that you can push model repositories on the Hub. This will allow you to save your trained models on the Hub so that you can share them with the community. Simply run the command:

git config --global credential.helper store
huggingface-cli login

And then enter an authentication token from https://huggingface.co/settings/tokens. Create a new token if you do not have one already. You should make sure that this token has "write" privileges.

You also have the option to configure Accelerate by running the following command. Note that you should set the number of GPUs you wish to use for training/inference, and also the data type (dtype) based on your device (e.g. bfloat16 on A100 GPUs, float16 on V100 GPUs, etc.):

accelerate config

Optionally, you can also login to Weights and Biases for automatic logging:

wandb login

Step 1: Create Text Descriptions

Creating text descriptions for the dataset comprises three sub-stages from DataSpeech, which we'll cover below.

1.A. Annotate the Expresso dataset

We'll use the main.py file from DataSpeech to label the following continuous variables:

• Speaking rate
• Signal-to-noise ratio (SNR)
• Reverberation
• Speech monotony

This can be done with the following command:

python ./dataspeech/main.py "ylacombe/expresso" \
  --configuration "default" \
  --text_column_name "text" \
  --audio_column_name "audio" \
  --cpu_num_workers 8 \
  --rename_column \
  --repo_id "expresso-tags"

Note that the script will be faster if you have GPUs at your disposal. It will automatically scale up to every GPU available in your environment. To control which GPUs to run the script on consider indicating via CUDA_VISIBLE_DEVICES environment variable.

The resulting dataset will be pushed to the Hugging Face Hub under your Hugging Face handle. Mine was pushed to