Speaker Diarization
Working out who spoke when in a recording with several speakers, including overlapping speech.
Tier Grades
Tier A: Frontier AI labs
feasible
NVIDIA built Sortformer to put speaker labels inside speech-to-text models.
Tier B: Indonesian AI labs
feasible
Public benchmarks such as the 9 used for the powerset loss allow national labs to train and test.
Tier C: Institution units such as ITB
feasible
The powerset loss came from a 2-author team, so diarization research fits a unit.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Open pipelines such as pyannote 3.1 can split meeting recordings by speaker in a minutes-taking pilot.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeNot in the sources read: the powerset and Sortformer abstracts give no compute.
- DataPowerset loss: 9 diarization benchmarks; TitaNet: DER 1.73% on AMI-MixHeadset.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 2 authors on the powerset paper; 9 on Sortformer.
- Time horizonMonths.
- Shelf-life signalsSpeech-to-text models that output speaker labels natively, as Sortformer aims for
- Pipeline roleTier A and Tier C release diarization models; Tier B tests them on Indonesian meetings; Tier D builds meeting minutes and call analytics with them.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
DiariZen (WavLM Large s80 md v2)speakerexemplar
pyannote speaker-diarization-3.1speakerexemplar
Sortformerspeakerexemplarmultitalker-parakeet-streaming-0.6b-v1speech recognitionproduced bypyannote segmentation-3.0speakerproduced byMSDD (multi-scale diarization decoder)speakerproduced by
MOSS-Transcribe-Diarizespeakerproduced bypyannote Community-1speakerproduced bypyannote Precision-2speakerproduced byLinks
R&D TopicSpeaker Diarization
ModelsSortformerNVIDIA, speakerpyannote speaker-diarization-3.1pyannoteAI, speakerDiariZen (WavLM Large s80 md v2)Brno University of Technology (BUT Speech@FIT), speakerMOSS-Transcribe-DiarizeOpenMOSS, speaker+5 more
SortformerNVIDIA, speakerpyannote speaker-diarization-3.1pyannoteAI, speakerDiariZen (WavLM Large s80 md v2)Brno University of Technology (BUT Speech@FIT), speakerMOSS-Transcribe-DiarizeOpenMOSS, speaker+5 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 9
multitalker-parakeet-streaming-0.6b-v1NVIDIA, speech recognitionproduced byDiariZen (WavLM Large s80 md v2)Brno University of Technology (BUT Speech@FIT), speakerexemplarMOSS-Transcribe-DiarizeOpenMOSS, speakerproduced byMSDD (multi-scale diarization decoder)NVIDIA, speakerproduced bypyannote Community-1pyannoteAI, speakerproduced bypyannote Precision-2pyannoteAI, speakerproduced bypyannote segmentation-3.0pyannoteAI, speakerproduced bypyannote speaker-diarization-3.1pyannoteAI, speakerexemplarSortformerNVIDIA, speakerexemplarPapers
- Fully Supervised Speaker DiarizationAonan Zhang et al., ICASSP 2019 (per arXiv comment), 2018 founding
- Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text SystemsTaejin Park et al., Proceedings of the 42nd International Conference on Machine Learning (ICML), 202, 2024 state of the art
- Powerset multi-class cross entropy loss for neural speaker diarizationAlexis Plaquet et al., INTERSPEECH 2023, Aug 2023, Dublin, Ireland. pp.3222-3226, 2023 state of the art
- End-to-End Neural Speaker Diarization with Self-attentionYusuke Fujita et al., ASRU 2019 (per arXiv comment), 2019 state of the art
- A Review of Speaker Diarization: Recent Advances with Deep LearningTae Jin Park et al., arXiv preprint, 2021 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2310.13025 https://arxiv.org/abs/2310.13025
- arxiv.org/abs/2409.06656 https://arxiv.org/abs/2409.06656
- arxiv.org/abs/2110.04410 https://arxiv.org/abs/2110.04410
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Speaker Diarization", https://aurion.semai.id/topics/speaker-diarization/, records of October 2026.