NVIDIA Nemotron 3 Diarization Tracks Eight Speakers in Audio

Automatic speech recognition tools excel at transcribing spoken words, yet transcription alone leaves critical conversational context unaddressed. Without knowing who spoke when, software cannot determine who made a commitment, raised an objection, or interrupted during a discussion. Speaker diarization provides the necessary timestamps for each participant, enabling downstream language models and analytics pipelines to attribute every phrase accurately. NVIDIA has expanded its speech ecosystem with the release of Nemotron 3 Diarization, an open-weight model with 100 million parameters designed to distinguish up to eight speakers across offline files and low-latency streaming audio.

Architecture and Arrival-Ordered Speaker Tracking

Nemotron 3 Diarization accepts 16 kHz single-channel audio across standard formats including .wav, .flac, .opus, and .mp3. The system converts raw audio into Mel-spectrogram features with a 10 ms frame step, which are subsequently stacked by a factor of eight to yield 80 ms encoder frames. These frames pass through a 31-layer Transformer encoder equipped with rotary positional embeddings (RoPE). A 1D convolutional layer then upsamples the encoder outputs back to a 10 ms temporal resolution, delivering a tensor of dimensions [T, 8] representing the activation probability for eight potential speaker channels at each time step.

This multichannel representation directly accommodates overlapping speech. If two participants talk simultaneously, two distinct channels register positive probabilities within the exact same frame. The architecture builds on the Sortformer methodology by ordering speakers strictly according to their arrival time. The first detected voice is assigned to channel one, the next to channel two, and subsequent voices fill remaining slots. This mechanism ensures stable anonymous speaker labeling throughout streaming sessions without requiring permutation re-evaluations across successive chunks.

• Arrival-Order Speaker Cache (AOSC): Retains historical speaker representations from previous audio chunks organized by channel.
• First-in, first-out (FIFO) queue: Supplies immediate preceding frame context directly to the active encoding window.
• Right context buffer: Ingests audio immediately following the current chunk to assist in accurate speaker boundary transitions.
• Anonymous identity assignment: Downstream systems map numerical channel outputs to actual individuals via active speaker verification or metadata.

Operating Latencies and Benchmark Performance

The model accommodates variable input-buffer latency settings calculated from the chunk size and right context multiplied by 80 ms. NVIDIA highlights four primary operational points: 30.4 seconds for offline workloads, alongside streaming buffers of 1.04 seconds, 0.64 seconds, and 0.32 seconds. While an extreme buffer of 80 ms is technically possible, NVIDIA identifies 0.32 seconds as the lowest recommended operating threshold for reliable speech processing. These metrics represent input audio buffering exclusively, excluding hardware execution, network transport, and automatic speech recognition computation.

On the initial Voice Arena Diarization-Bench evaluation across 139 English conversations totaling approximately 22 hours, Nemotron 3 Diarization achieved first place among 12 systems and 17 configurations. Scoring under conditions with overlapping speech, system-generated speech activity detection, and a zero collar, the model recorded a 14.72% diarization error rate (DER). This represents an approximate 24% relative reduction compared to the 19.3% DER of the runner-up system, while also leading with 100 ms and 250 ms collars across online and in-person audio. NVIDIA notes that these benchmark figures remain subject to revision upon completion of Voice Arena's final Version 1 review.

When compared against NVIDIA's previous 4-speaker checkpoint (diar_streaming_sortformer_4spk-v2.1) at a 1.04-second latency point, Nemotron 3 Diarization reduced DER across all eight tested conditions, with relative improvements ranging from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM. The unweighted mean relative DER reduction across these eight benchmarks reached 41.0%. One isolated regression occurred on 2-speaker CALLHOME at 30.4 seconds latency, where DER rose from 5.68% to 5.98%, though the complete CALLHOME-Part2 evaluation improved from 10.32% to 9.10%. Furthermore, compiled batch-32 throughput on an NVIDIA RTX PRO 5000 GPU using BF16 precision reached 15,113x real-time factor (RTFx) at 30.4 seconds, up from 2,619x for the baseline.

Training Data, Deployment, and Practical Constraints

Training the 100M-parameter architecture required combining roughly 10,000 hours of real conversational audio with 82,611 hours of simulated multi-talker mixtures. The simulated dataset drew from licensed source audio spanning 21 languages, alongside real-world multi-speaker recordings licensed from David AI. Incorporating the David AI corpora decreased the compound diarization error rate from 11.19% to 10.42% across offline and ultra-low-latency evaluations.

Nemotron 3 Diarization is distributed under the OpenMDW License 1.1 on Hugging Face, authorizing commercial use. The model runs within the NVIDIA NeMo Speech framework on Linux systems equipped with NVIDIA Ampere, Ada Lovelace, Hopper, or Blackwell GPUs, requiring Python 3.12 or newer. For end-to-end multi-speaker transcription, the model pairs with speech recognition models such as Parakeet TDT 0.6B v3. Deployment is currently supported through platforms like Baseten and DigitalOcean, with on-device integration enabled by the Argmax Pro SDK 3, while Hugging Face Inference Providers are not yet supported.

Organizations deploying the system must account for intrinsic architectural constraints. The model strictly caps simultaneous speaker tracking at eight individuals; conversations exceeding this threshold result in missed speech or misattributed channels. In addition, severe acoustic reverberation, heavy background noise, far-field microphone capture, and domain distribution shifts can elevate diarization error rates.

Implications for Enterprise Voice Pipelines

For engineering teams developing conversational intelligence, call center analytics, and real-time meeting assistants, Nemotron 3 Diarization provides a commercially viable open-weight component capable of parsing complex overlapping dialogue. By doubling the concurrent speaker threshold from four to eight and maintaining consistent channel assignments in streaming mode, organizations can build responsive transcription systems that attribute spoken commitments accurately without relying on proprietary black-box diarization APIs.