parler-tts-mini-multilingual

Parler-TTS Mini Multilingual v1.1 is a multilingual extension of Parler-TTS Mini.

🚨 As compared to Mini Multilingual v1, this version was trained with some consistent speaker names and with better format for descriptions. 🚨

It is a fine-tuned version, trained on a cleaned version of CML-TTS and on the non-English version of Multilingual LibriSpeech. In all, this represents some 9,200 hours of non-English data. To retain English capabilities, we also added back the LibriTTS-R English dataset, some 580h of high-quality English data.

Parler-TTS Mini Multilingual can speak in 8 European languages: English, French, Spanish, Portuguese, Polish, German, Italian and Dutch.

Thanks to its better prompt tokenizer, it can easily be extended to other languages. This tokenizer has a larger vocabulary and handles byte fallback, which simplifies multilingual training.

🚨 This work is the result of a collaboration between the HuggingFace audio team and the Quantum Squadra team. The AI4Bharat team also provided advice and assistance in improving tokenization. 🚨

📖 Quick Index

• 👨‍💻 Installation
• 🎲 Using a random voice
• 🎯 Using a specific speaker
• Motivation
• Optimizing inference

🛠️ Usage

🚨Unlike previous versions of Parler-TTS, here we use two tokenizers - one for the prompt and one for the description.🚨

👨‍💻 Installation

Using Parler-TTS is as simple as "bonjour". Simply install the library once:

pip install git+https://github.com/huggingface/parler-tts.git

🎲 Random voice

Parler-TTS Mini Multilingual has been trained to generate speech with features that can be controlled with a simple text prompt, for example:

import torch
from parler_tts import ParlerTTSForConditionalGeneration
from transformers import AutoTokenizer
import soundfile as sf

device = "cuda:0" if torch.cuda.is_available() else "cpu"

model = ParlerTTSForConditionalGeneration.from_pretrained("parler-tts/parler-tts-mini-multilingual-v1.1").to(device)
tokenizer = AutoTokenizer.from_pretrained("parler-tts/parler-tts-mini-multilingual-v1.1")
description_tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder._name_or_path)

prompt = "Salut toi, comment vas-tu aujourd'hui?"
description = "A female speaker delivers a slightly expressive and animated speech with a moderate speed and pitch. The recording is of very high quality, with the speaker's voice sounding clear and very close up."

input_ids = description_tokenizer(description, return_tensors="pt").input_ids.to(device)
prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)

generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)
audio_arr = generation.cpu().numpy().squeeze()
sf.write("parler_tts_out.wav", audio_arr, model.config.sampling_rate)

🎯 Using a specific speaker

To ensure speaker consistency across generations, this checkpoint was also trained on 16 speakers, characterized by name (e.g. Daniel, Christine, Richard, Nicole, ...).

To take advantage of this, simply adapt your text description to specify which speaker to use: Daniel's voice is monotone yet slightly fast in delivery, with a very close recording that almost has no background noise.

import torch
from parler_tts import ParlerTTSForConditionalGeneration
from transformers import AutoTokenizer
import soundfile as sf

device = "cuda:0" if torch.cuda.is_available() else "cpu"

model = ParlerTTSForConditionalGeneration.from_pretrained("parler-tts/parler-tts-mini-multilingual-v1.1").to(device)
tokenizer = AutoTokenizer.from_pretrained("parler-tts/parler-tts-mini-multilingual-v1.1")
description_tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder._name_or_path)

prompt = "Salut toi, comment vas-tu aujourd'hui?"
description = "Daniel's voice is monotone yet slightly fast in delivery, with a very close recording that almost has no background noise."

input_ids = description_tokenizer(description, return_tensors="pt").input_ids.to(device)
prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)

generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)
audio_arr = generation.cpu().numpy().squeeze()
sf.write("parler_tts_out.wav", audio_arr, model.config.sampling_rate)

You can choose a speaker from this list: