GigaChat3.1-Audio

GigaChat Audio 10B is an audio-native LLM built on top of the GigaChat 3.1 Lightning text model. A Conformer speech encoder and a modality adapter feed audio embeddings directly into a Mixture-of-Experts decoder, so the model keeps the text quality of its base while adding speech understanding.

Capabilities: audio question answering and classification, temporal grounding (localization in long audio, timestamped event descriptions, audio summarization with timestamps), tool-use, and text-only tasks.

The temporal grounding skills are trained on TimeGround-1M — a purpose-built dataset of long-form audio paired with time-aligned annotations.

Evaluation

1. Core audio tasks vs open models

2. Timing tasks (detailed)

TL — temporal localization: find when something is discussed (mIoU vs the reference span). TD — timestamped descriptions of audio events (overall grade 0–5). SUM — long-audio summarization with timestamps; single score = mean of factual accuracy, timing structure and audio coverage with timestamps.

3. Text quality vs the text base model

Adding the audio modality shifts text quality: some benchmarks regress (MMLU-Pro, IFEval-ru, BBH), others improve (RuBQ, GPQA Diamond).

Quickstart (Transformers)

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

model_name = "ai-sage/GigaChat3.1-Audio-10B-A1.8B"
processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda:0",
)

messages = [{"role": "user", "content": [
    {"type": "audio", "path": "90min_lecture.wav"},
    {"type": "text", "text": "When does the speaker mention Star Wars midi-chlorians?"},
]}]
inputs = processor.prepare_for_inference(messages, device=model.device)
output_ids = model.generate(**inputs, max_new_tokens=256)
answer_ids = output_ids[0, inputs["input_ids"].shape[1]:]
print(processor.decode(answer_ids))
# > ... in the interval 01:27:46 to 01:27:53. In this segment they explain ...

vLLM (single GPU, native multimodal)

Both encoder and decoder run inside vLLM. compilation_config disables torch.compile (needed for the Conformer tower); max_num_batched_tokens sizes the audio encoder cache (~90 min is 33k tokens):

import librosa
import os
from transformers import AutoProcessor
from vllm import LLM, SamplingParams

os.environ.setdefault("VLLM_WORKER_MULTIPROC_METHOD", "spawn")

model_name = "ai-sage/GigaChat3.1-Audio-10B-A1.8B"

def main():
    processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
    llm = LLM(
        model=model_name,
        trust_remote_code=True,
        max_model_len=65536,
        max_num_batched_tokens=65536,
        limit_mm_per_prompt={"audio": 1},
        compilation_config={"mode": 0, "cudagraph_mode": "FULL"},
    )

    messages = [{"role": "user", "content": [
        {"type": "audio", "path": "90min_lecture.wav"},
        {"type": "text", "text": "When does the speaker mention Star Wars midi-chlorians?"},
    ]}]
    text, audio_paths = processor.render_prompt(messages)
    wav, _ = librosa.load(audio_paths[0], sr=16000, mono=True)
    out = llm.generate(
        {"prompt": text, "multi_modal_data": {"audio": [(wav, 16000)]}},
        SamplingParams(temperature=0.0, max_tokens=256),
    )
    print(processor.decode(out[0].outputs[0].token_ids))
    # > ... in the interval 01:27:46 to 01:27:53. In this segment they explain ...

if __name__ == "__main__":
    main()

Throughput

Single-request decode speed on one H100 (vLLM 0.18.0, bf16, greedy), by amount of audio held in context: