Llama-OuteTTS-1.0

OuteTTS Version 1.0

This update brings significant improvements in speech synthesis and voice cloning—delivering a more powerful, accurate, and user-friendly experience in a compact size.

What's New

1. Prompt Revamp & Dependency Removal

• Automatic Word Alignment: The model now performs word alignment internally. Simply input raw text—no pre-processing required—and the model handles the rest, streamlining your workflow. For optimal results, use normalized, readable text without newlines (light normalization is applied automatically in outetts library).
• Native Multilingual Text Support: Direct support for native text across multiple languages eliminates the need for romanization.
• Enhanced Metadata Integration: The updated prompt system incorporates additional metadata (time, energy, spectral centroid, pitch) at both global and word levels, improving speaker flow and synthesis quality.
• Special Tokens for Audio Codebooks: New tokens for c1 (codebook 1) and c2 (codebook 2).

2. New Audio Encoder Model

• DAC Encoder: Integrates a DAC audio encoder from ibm-research/DAC.speech.v1.0, utilizing two codebooks for high quality audio reconstruction.
• Performance Trade-off: Improved audio fidelity increases the token generation rate from 75 to 150 tokens per second. This trade-off prioritizes quality, especially for multilingual applications.

3. Voice Cloning

• One-Shot Voice Cloning: To achieve one-shot cloning, the model typically requires only around 10 seconds of reference audio to produce an accurate voice representation.
• Improved Accuracy: Enhanced by the new encoder and additional training metadata, voice cloning is now more natural and precise.

4. Auto Text Alignment & Numerical Support

• Automatic Text Alignment: Aligns raw text at the word level, even for languages without clear boundaries (e.g., Japanese, Chinese), using insights from pre-processed training data.
• Direct Numerical Input: Built-in multilingual numerical support allows direct use of numbers in prompts—no textual conversion needed. (The model typically chooses the dominant language present. Mixing languages in a single prompt may lead to mistakes.)

5. Multilingual Capabilities

• Supported Languages: OuteTTS offers varying proficiency levels across languages, based on training data exposure.

• High Training Data Languages: These languages feature extensive training: English, Arabic, Chinese, Dutch, French, German, Italian, Japanese, Korean, Lithuanian, Russian, Spanish

• Moderate Training Data Languages: These languages received moderate training, offering good performance with occasional limitations: Portuguese, Belarusian, Bengali, Georgian, Hungarian, Latvian, Persian/Farsi, Polish, Swahili, Tamil, Ukrainian

• Beyond Supported Languages: The model can generate speech in untrained languages with varying success. Experiment with unlisted languages, though results may not be optimal.

Video Showcase

Your browser does not support the video tag.

Quick Start Guide

Getting started with OuteTTS is simple:

Installation

🔗 Installation instructions

Basic Usage

import outetts

# Initialize the interface
interface = outetts.Interface(
    config=outetts.ModelConfig.auto_config(
        model=outetts.Models.VERSION_1_0_SIZE_1B,
        # For llama.cpp backend
        backend=outetts.Backend.LLAMACPP,
        quantization=outetts.LlamaCppQuantization.FP16
        # For transformers backend
        # backend=outetts.Backend.HF,
    )
)

# Load the default speaker profile
speaker = interface.load_default_speaker("EN-FEMALE-1-NEUTRAL")

# Or create your own speaker profiles in seconds and reuse them instantly
# speaker = interface.create_speaker("path/to/audio.wav")
# interface.save_speaker(speaker, "speaker.json")
# speaker = interface.load_speaker("speaker.json")

# Generate speech
output = interface.generate(
    config=outetts.GenerationConfig(
        text="Hello, how are you doing?",
        generation_type=outetts.GenerationType.CHUNKED,
        speaker=speaker,
        sampler_config=outetts.SamplerConfig(
            temperature=0.4
        ),
    )
)

# Save to file
output.save("output.wav")

More Configuration Options

For advanced settings and customization, visit the official repository: 🔗 interface_usage.md

Usage Recommendations

Speaker Reference

The model is designed to be used with a speaker reference. Without one, it generates random vocal characteristics, often leading to lower-quality outputs. The model inherits the referenced speaker's emotion, style, and accent. When transcribing to other languages with the same speaker, you may observe the model retaining the original accent.

Multilingual Application

It is recommended to create a speaker profile in the language you intend to use. This helps achieve the best results in that specific language