midashenglm-gen

MiDashengLM-Gen

English | 中文

MiDashengLM-Gen is an end-to-end framework that uses a pre-trained Large Language Model and audio tokenizer as the backbone, combined with per-token conditional flow matching for autoregressive, variable-length mixed-audio scene generation. It generates coherent 16 kHz audio scenes that simultaneously blend speech, music, sound effects, and environmental acoustics from structured text descriptions.

Architecture

Left: training pipeline with flow matching loss. Right: autoregressive inference pipeline.

Key components:

Input Format

Input uses structured multi-view captions with special tokens to describe different aspects of an audio scene. Use `` for absent elements.

Installation

pip install torch torchaudio "transformers>=4.51" einops safetensors soundfile tqdm numpy x-transformers

Quick Start

from transformers import AutoModel
import soundfile as sf

model = AutoModel.from_pretrained("mispeech/midashenglm-gen", trust_remote_code=True)
model = model.cuda()

result = model.generate(
    " A comedian delivering a punchline followed by uproarious crowd laughter and an upbeat jazz band hit "
    " And that is why I never buy cheap luggage anymore! "
    " expressive comedic male voice "
    " sudden upbeat jazz band sting "
    " uproarious crowd laughter "
    " intimate comedy club"
)

sf.write("output.wav", result["audio"], result["sample_rate"])

Batch Generation

texts = [
    " A comedian delivering a punchline followed by uproarious crowd laughter and an upbeat jazz band hit  And that is why I never buy cheap luggage anymore!  expressive comedic male voice  sudden upbeat jazz band sting  uproarious crowd laughter  intimate comedy club",
    " Upbeat instrumental music with occasional whistling and low audio quality.      Intermittent whistling emerges in later segment alongside persistent instrumental playback.  Bright melodic composition with guitar, piano, and rhythmic percussion elements, maintaining a consistently upbeat character.  Low-fidelity recording with compressed dynamics and limited frequency response.",
]
result = model.generate(texts)
for i, audio in enumerate(result["audio"]):
    sf.write(f"output_{i}.wav", audio, result["sample_rate"])

Generation Parameters

result = model.generate(
    "...",
    eval_cfg=2.0,           # Classifier-free guidance scale
    stop_threshold=0.5,     # Stop prediction threshold
    min_stop_step=5,        # Minimum steps before stopping
    seed=42,                # Random seed for reproducibility
)

Citation

@article{sun2026midashenglmgen,
  title={MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching},
  author={Sun, Xingwei and Dinkel, Heinrich and Li, Gang and Mei, Jiahao and Niu, Yadong and Han, Zerui and Jiang, Yuepeng and Zhou, Jiahao and Fan, Lichun and Luan, Jian},
  journal={arXiv preprint},
  eprint={2608.11804},
  archivePrefix={arXiv},
  year={2026}
}

License

Apache 2.0

Use Restrictions

You are solely responsible for your use of MiDashengLM-Gen and any outputs, actions, or consequences arising therefrom, and you agree not to use MiDashengLM-Gen or any derivatives thereof:

• For any unlawful, fraudulent, or malicious purpose, or in any manner that violates any applicable laws or regulations;
• To infringe upon the intellectual property rights, privacy rights, publicity rights, or other lawful rights or interests of any third party;
• To exploit, harm, harass, defame, unlawfully discriminate against, or otherwise adversely affect any individual or group, including minors or vulnerable persons;
• For any military purpose or application.