diar_streaming_sortformer_4spk

About

What diar_streaming_sortformer_4spk is

Streaming Sortformer Diarizer 4spk v2.1 img { display: inline; } [](#model-architecture) This model is a streaming version of Sortformer diarizer. Sortformer[1] is a novel end-to-end neural model for speaker diarization, trained with unconventional objectives compared to existing end-to-end diarization models. Streaming Sortformer[2] employs an Arrival-Order Speaker Cache (AOSC) to store frame-level acoustic embeddings of previously observed speakers. Sortformer resolves permutation problem in diarization following the arrival-time order of the speech segments from each speaker. This speaker diarization model can be used to enable the NeMo Voice Agent to recognize speakers in conversations. See the NeMo Voice Agent and the YAML configuration for more details. Discover more from NVIDIA: For documentation, deployment guides, enterprise-ready APIs, and the latest open models—including Nemotron and other cutting-edge speech, translation, and generative AI—visit the NVIDIA Developer Portal at developer.nvidia.com. Join the community to access tools, support, and resources to accelerate your development with NVIDIA’s NeMo, Riva, NIM, and foundation models.<br Explore more from NVIDIA: <br What is Nemotron?<br NVIDIA Developer Nemotron<br NVIDIA Riva Speech<br NeMo Documentation<br Model Architecture Streaming sortformer employs pre-encode layer in the Fast-Conformer to generate speaker-cache. At each step, speaker cache is filtered to only retain the high-quality speaker cache vectors. Aside from speaker-cache management part, streaming Sortformer follows the architecture of the offline version of Sortformer. Sortformer consists of an L-size (17 layers) [NeMo Encoder for Speech Tasks (NEST)](https://arxiv.org/abs/2408.13106)[3] which is based on Fast-Conformer[4] encoder. Following that, an 18-layer Transformer[5] encoder with hidden size of 192, and two feedforward layers with 4 sigmoid outputs for each frame input at the top layer. More information can be found in the Streaming Sortformer paper[2]. NVIDIA NeMo To train, fine-tune or perform diarization with Sortformer, you will need to install NVIDIA NeMo[6]. We recommend you install it after you've installed Cython

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How audio & music work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Benchmarks

Reported results

As published on the model card — the maker's own numbers, not measured by AxForge.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys diar-streaming-sortformer-4spk for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (diar-streaming-sortformer-4spk below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/audio/transcriptions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -F model="diar-streaming-sortformer-4spk" -F file=@audio.mp3

Details

Languages, data & research

Papers

Licence

Commercial licence needed

Sources

Weights & code

Explore

More audio & music