SALMONN-2

Results

SALMONN-2 achieves strong performance on three audio-language model (ALLM) benchmarks while using 18.2k hours of supervised audio-text training data. The table below shows the 8B model comparison.

Minimal Hugging Face inference

The following example loads this repository directly with Hugging Face transformers; cloning or installing the SALMONN-2 GitHub package is not required.

pip install "transformers>=4.57,<5" accelerate torch torchaudio

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "marcoyang/salmonn-2-8b-test"

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
    fix_mistral_regex=False,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
).eval()
processor.prepare_model(model)

inputs = processor(
    audios=["example.wav"],
    instruction="Please describe the audio.",
)
device = next(model.parameters()).device

with torch.inference_mode():
    output_ids = model.generate(
        **inputs.to(device),
        max_new_tokens=256,
        do_sample=False,
    )

print(processor.decode(output_ids[0]))

The processor loads and resamples audio, computes the SPEAR filterbank input, formats the chat prompt, and returns all model inputs. The example uses bfloat16 and automatic device placement. A CUDA GPU is recommended for practical inference.

Contextual ASR

For text-only contextual words, pass a list of strings:

inputs = processor(
    audios=["main_utterance.wav"],
    instruction="Recognize the speech and give me the transcription.",
    context=["howes", "wszelaki"],
)

To provide both the spelling and pronunciation of each contextual word, pair the text with a context audio file:

inputs = processor(
    audios=["main_utterance.wav"],
    instruction="Recognize the speech and give me the transcription.",
    context=[
        {"text": "howes", "audio": "howes.wav"},
        {"text": "wszelaki", "audio": "wszelaki.wav"},
    ],
)

The processor places all model-specific audio markers and contextual formatting automatically.

Advanced prompt placement

For custom multi-audio layouts, use formatted_prompt with one `` marker per input file. Audio files are matched to the markers from left to right:

inputs = processor(
    audios=["main.wav", "example.wav"],
    formatted_prompt=(
        "Compare the main recording with this example: "
        "What do they have in common?"
    ),
)

The processor still loads the audio, computes filterbanks, validates marker alignment, applies the chat template, and returns model-ready tensors.

License

Apache License 2.0. See LICENSE.

Citation

@inproceedings{yang2026spear,
  title     = {SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations},
  author    = {Yang, Xubo and Yang, Yuxuan and Jin, Ziyang and Cui, Zeyu and Wu, Wen and Li, Bo and Zhang, Chao and Woodland, Philip C.},
  booktitle = {Proceedings of the Forty-third International Conference on Machine Learning},
  year      = {2026}
}