Model reference · open weights
multi-talker-whisper-small-ami is an open-weight audio or speech model from espnet. multi-talker-whisper-small-ami (BF16) weighs 967 MB; the smallest configuration that runs it is RTX 3060 12 GB.
Summary of the espnet/multi-talker-whisper-small-ami model card, 2026-10-01
What it is
| Released by | espnet |
|---|---|
| Released | 2026-05-24 |
| VRAM | 967 MB for the weights |
What it runs on
| Card | One stream | Memory |
|---|---|---|
| RTX 3060 12 GB | fits | 11.6 GB |
| RTX 4060 Ti 16 GB | fits | 15.4 GB |
| RTX 3090 24 GB | fits | 23.4 GB |
| RTX 4090 24 GB | fits | 23.4 GB |
| RTX 5090 32 GB | fits | 31.0 GB |
| L40S 48 GB | fits | 44.0 GB |
| A100 80 GB | fits | 78.2 GB |
| H100 80 GB | fits | 78.1 GB |
| RTX PRO 6000 Blackwell 96 GB | fits | 93.8 GB |
| DGX Spark (GB10) 128 GB unified | fits | 107 GB |
| H200 141 GB | fits | 138 GB |
| B200 180 GB | fits | 176 GB |
From the model card
ESPnet checkpoint for serialized-output-training (SOT) multi-talker ASR on
the AMI meeting corpus, built on top of openai/whisper-small. Trained to
emit a single transcript containing every speaker in FIFO order, separated
by a `` speaker-change token, with per-speaker Whisper-style timestamps.
| File | Purpose |
|---|---|
model.pth | ESPnet-format weights (479 keys) |
config.yaml | Model architecture / preprocessor spec |
token_list.txt | 51,865-token Whisper multilingual vocabulary |
This checkpoint is consumed by the
egs2/ami/sot_asr1
recipe. After cloning ESPnet and preparing the AMI test data, place the
three files under exp/whisper-sot-small-ami/ and run:
./run.sh --released_model exp/whisper-sot-small-ami \
--whisper_model small \
--decode_test_sets test
The recipe prints utterance-group cpWER and utterance-group DER under
exp/whisper-sot-small-ami/decode_released/test/eval/.
| overall | 1-spk | 2-spk | 3-spk | 4-spk |
|---|---|---|---|---|
| 27.95 | 15.36 | 25.54 | 38.94 | 52.44 |
| overall | 1-spk | 2-spk | 3-spk | 4-spk |
|---|---|---|---|---|
| 9.84 | 1.47 | 6.99 | 18.65 | 29.43 |
Quoted from the model card on Hugging Face. The full card is behind the Hugging Face link above.
How it works