glm-4-voice-of-reason-stitch

Run it on a wav

The audio front end is GLM-4-Voice's, unchanged and not in this repo, so clone it for the speech tokenizer:

git clone https://github.com/THUDM/GLM-4-Voice
uv init glm-of-reason && cd glm-of-reason
uv add "transformers>=4.44,<4.48" torch torchaudio accelerate tiktoken soundfile

tiktoken is needed by the tokenizer's remote code and soundfile by torchaudio to read your file; both fail late and unhelpfully if missing. Save the script below as demo.py, point AUDIO at your wav — anything torchaudio can read, any sample rate — and run uv run demo.py:

import re, sys, torch
sys.path.insert(0, "../GLM-4-Voice")           # the clone, next to the project

from transformers import AutoModel, AutoTokenizer, WhisperFeatureExtractor
from speech_tokenizer.modeling_whisper import WhisperVQEncoder
from speech_tokenizer.utils import extract_speech_token

AUDIO = "question.wav"                                    # <-- your spoken question
REPO = "kyutai/glm-4-voice-of-reason-stitch-9b"

SYSTEM = (
    "User will provide you with a speech instruction. Do it step by step. "
    "First think in partial reasoning chunks of 100 tokens using [SOPR] and [EOPR], "
    "then respond in an interleaved manner, with 13 text tokens followed by 26 audio tokens."
)

tokenizer = AutoTokenizer.from_pretrained(REPO, trust_remote_code=True)
model = AutoModel.from_pretrained(
    REPO, torch_dtype=torch.bfloat16, device_map="cuda", trust_remote_code=True
).eval()

whisper = WhisperVQEncoder.from_pretrained("THUDM/glm-4-voice-tokenizer").eval().to("cuda")
features = WhisperFeatureExtractor.from_pretrained("THUDM/glm-4-voice-tokenizer")

audio_tokens = extract_speech_token(whisper, features, [AUDIO])[0]
user = "" + "".join(f"" for t in audio_tokens) + ""
prompt = f"\n{SYSTEM}\n{user}streaming_transcription\n"

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
out = model.generate(
    **inputs,
    max_new_tokens=1000,
    do_sample=False,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.convert_tokens_to_ids(""),
)
new = out[0, inputs.input_ids.shape[1]:].tolist()

audio_offset = tokenizer.convert_tokens_to_ids("")
text = tokenizer.decode([t for t in new if t < audio_offset], skip_special_tokens=True)
speech_tokens = [t - audio_offset for t in new if t >= audio_offset]

print(text)                                               # reasoning and answer, interleaved
print(re.sub(r"\[SOPR\].*?(\[EOPR\]|$)", "", text, flags=re.S))   # only what is spoken

text comes out with the unspoken reasoning wrapped in [SOPR] ... [EOPR]; everything outside those markers is the spoken answer, which is why the second print is the one to read aloud. For example:

[SOPR]A navarin is a traditional dish ... the type of meat used in a navarin is lamb. The answer
is[EOPR]A navarin is a Middle Eastern dish made with lamb, so[SOPR] lamb.[EOPR] the meat used is lamb.

speech_tokens are the audio codes of that same answer; feed them to THUDM/glm-4-voice-decoder to get a waveform, or run the full duplex demo from the GLM-4-Voice repo with --model-path pointing here.

This script was run as printed, on one H100, with transformers 4.47.1 and torch 2.8.0. Pick a torch build that matches your driver: the newest wheel needs a newer CUDA than many clusters run.

License

Inherited from GLM-4-Voice; see the license link above.