MOSS-TTS-Local-Transformer

MOSS-TTS-Local-Transformer-v1.5

MOSS-TTS-Local-Transformer-v1.5 is continued from MOSS-TTS-Local-Transformer-v1.0. It preserves the main 1.0 capabilities, including zero-shot voice cloning, long-form speech generation, token-level duration control, Pinyin/IPA pronunciation control, multilingual synthesis, and code-switching. For the full 1.0 feature walkthrough, input schema, and evaluation tables, please refer to the MOSS-TTS-Local-Transformer-v1.0 README.

Compared with MOSS-TTS-Local-Transformer-v1.0, v1.5 focuses on the following improvements:

• Higher-fidelity stereo audio modeling: v1.5 uses MOSS-Audio-Tokenizer-v2 as the audio tokenizer, supporting native 48 kHz stereo input and output for richer spatial detail and more natural perceived audio quality. Since the codec output is stereo, save the [channels, samples] tensor returned by processor.decode(...) directly.
• Stronger multilingual synthesis with language tags: when the language field is omitted, v1.5 may improve some languages and regress slightly on others compared with 1.0. When the language is specified, v1.5 is stronger than 1.0 on almost all supported languages. Set the tag when building the user message, for example processor.build_user_message(text=text_fr, language="French").
• More stable voice cloning: v1.5 improves speaker similarity and reduces cloning variance, making repeated generations more consistent.
• Better long-reference, short-text cloning: v1.5 handles scenarios where the reference audio is much longer than the target text more reliably than 1.0.
• More stable punctuation-following prosody: v1.5 follows punctuation-driven pauses more closely, especially in long sentences.
• Explicit pause control: v1.5 supports inline pause markers such as "[pause 3.2s]". For example, 我今天学习了一首中国的古诗，它的名字是[pause 3.2s]静夜思！ inserts an explicit 3.2s pause before 静夜思.

Supported Languages

MOSS-TTS Local Transformer v1.5 supports 31 languages. It keeps the 20 languages supported by MOSS-TTS-Local-Transformer-v1.0 and extends multilingual continued training to additional languages including Cantonese, Dutch, Finnish, Hindi, Macedonian, Malay, Romanian, Swahili, Tagalog, Thai, and Vietnamese.

Quick Start

Environment Setup

We recommend a clean, isolated Python environment with Transformers 5.0.0, or a recent Transformers version with Qwen3 support, to avoid dependency conflicts.

conda create -n moss-tts python=3.12 -y
conda activate moss-tts

Install all required dependencies:

git clone https://github.com/OpenMOSS/MOSS-TTS.git
cd MOSS-TTS
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime]"

(Optional) Install FlashAttention 2

For better speed and lower GPU memory usage, you can install FlashAttention 2 if your hardware supports it.

pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[flash-attn]" --no-build-isolation

If your machine has limited RAM and many CPU cores, you can cap build parallelism:

MAX_JOBS=4 pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[flash-attn]" --no-build-isolation

Notes:

• Dependencies are managed in pyproject.toml, which currently pins torch==2.9.1+cu128 and torchaudio==2.9.1+cu128.
• If FlashAttention 2 fails to build on your machine, you can skip it and use the default attention backend.
• FlashAttention 2 is only available on supported GPUs and is typically used with torch.float16 or torch.bfloat16.

Basic Usage

Tip: MOSS-TTS-Local-Transformer-v1.5 uses a fixed 12-codebook RVQ depth. Do not set n_vq_for_inference to a value different from config.n_vq.

MOSS-TTS-Local-Transformer-v1.5 provides the standard Hugging Face AutoProcessor and AutoModel interface. The examples below cover:

1. Direct generation with language tags
2. Voice cloning
3. Duration control
4. Explicit pause control with [pause X.Ys]

from pathlib import Path
from tqdm import tqdm
import importlib.util

import torch
import torchaudio
from transformers import AutoModel, AutoProcessor

# Disable the broken cuDNN SDPA backend on some CUDA/PyTorch combinations.
torch.backends.cuda.enable_cudnn_sdp(False)
# Keep these enabled as fallbacks.
torch.backends.cuda.enable_flash_sdp(True)
torch.backends.cuda.enable_mem_efficient_sdp(True)
torch.backends.cuda.enable_math_sdp(True)

pretrained_model_name_or_path = "OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

def resolve_attn_implementation() -> str:
    # Prefer FlashAttention 2 when package + device conditions are met.
    if (
        device == "cuda"
        and importlib.util.find_spec("flash_attn") is not None