Model reference · open weights
diar_sortformer_4spk is an open-weight audio or speech model from nvidia, listed in the AxForge catalogue. AxForge can bring it up on EU-owned hardware for you on request — with the licence handled where one is required.
About
A newer streaming Sortformer is available at huggingface.co/nvidia/diarstreamingsortformer4spk-v2. Sortformer Diarizer 4spk v1 img { display: inline; } [](#model-architecture) Sortformer[1] is a novel end-to-end neural model for speaker diarization, trained with unconventional objectives compared to existing end-to-end diarization models. Sortformer resolves permutation problem in diarization following the arrival-time order of the speech segments from each speaker. Discover more from NVIDIA: For documentation, deployment guides, enterprise-ready APIs, and the latest open models—including Nemotron and other cutting-edge speech, translation, and generative AI—visit the NVIDIA Developer Portal at developer.nvidia.com. Join the community to access tools, support, and resources to accelerate your development with NVIDIA’s NeMo, Riva, NIM, and foundation models.<br Explore more from NVIDIA: <br What is Nemotron?<br NVIDIA Developer Nemotron<br NVIDIA Riva Speech<br NeMo Documentation<br Model Architecture Sortformer consists of an L-size (18 layers) [NeMo Encoder for Speech Tasks (NEST)](https://arxiv.org/abs/2408.13106)[2] which is based on Fast-Conformer[3] encoder. Following that, an 18-layer Transformer[4] encoder with hidden size of 192, and two feedforward layers with 4 sigmoid outputs for each frame input at the top layer. More information can be found in the Sortformer paper[1]. NVIDIA NeMo To train, fine-tune or perform diarization with Sortformer, you will need to install NVIDIA NeMo[5]. We recommend you install it after you've installed Cython and latest PyTorch version. How to Use this Model The model is available for use in the NeMo Framework[5], and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset. Loading the Model Input Format Input to Sortformer can be an individual audio file: or a list of paths to audio files: or a jsonl manifest file: where each line is a dictionary containing the following fields: Getting Diarization Results To perform speaker diarization and get a list of speaker-marked speech segments in the format 'beginseconds, endseconds, speakerindex', simply use: To obtain tensors of speaker activ
Summarised from the published model card. Read the full card on the HuggingFace links below.
Specifications
| Maker | nvidia |
|---|---|
| Type | Audio & music |
| Parameters (lead) | 124M |
| Variants | 1 |
| Runs with | nemo |
| Released | 2024-12-09 |
| Popularity | 173k downloads / month |
| Likes | 151 |
| Licence | Commercial licence needed |
How it works
Variants
Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.
| Variant | Params | Precision | VRAM | Fits 16 GB | Weights |
|---|---|---|---|---|---|
| diar_sortformer_4spk-v1 | 124M | BF16 | ~0.3 GB | ✓ | Weights ↗ |
Benchmarks
As published on the model card — the maker's own numbers, not measured by AxForge.
| Task | Dataset | Metric | Score |
|---|---|---|---|
| Speaker Diarization | DIHARD3-eval | Test DER | 14.76 |
| Speaker Diarization | CALLHOME (NIST-SRE-2000 Disc8) | Test DER | 5.85 |
| Speaker Diarization | CALLHOME (NIST-SRE-2000 Disc8) | Test DER | 8.46 |
| Speaker Diarization | CALLHOME (NIST-SRE-2000 Disc8) | Test DER | 12.59 |
| Speaker Diarization | call_home_american_english_speech | Test DER | 6.86 |
Using it via the API
Once AxForge deploys diar-sortformer-4spk for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (diar-sortformer-4spk below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/audio/transcriptions \ -H "Authorization: Bearer $AXFORGE_API_KEY" \ -F model="diar-sortformer-4spk" -F file=@audio.mp3
Details
Trained / evaluated on
Tags
Papers
Licence
The weights are open but cc-by-nc-4.0 needs a commercial agreement for business use. AxForge can arrange that licence and host the model for you — you pay AxForge, we settle with the model’s maker. Ask us for a quote. Read the licence ↗