IDSV (Indonesian Speech Variability)
Indonesian ASR WER/CER by speaking style (read or spontaneous, formal or informal, clean or noisy)
Southeast Asian and Indonesianstatic
Score in WER, Table 3
Definition
- Version
- not in the notes
- Category
- Southeast Asian and Indonesian
- Maintainer
- Institut Teknologi Bandung (ITB)
- Status
- static
- Home page
- https://arxiv.org/html/2410.08828v2
Sources
- Enhancing Indonesian Automatic Speech Recognition: Evaluating Multilingual Models with Diverse Speech Variabilities arxiv.org https://arxiv.org/html/2410.08828v2
Scores at a Glance
Score in WER, Table 3
Lower is better for this unit.
1FT-whisper (IDSV)Institut Teknologi Bandung (ITB), Universitas Indonesia, JAIST, NAISTIDSV test, all groupsMMS fine-tuned with LM (IDSV)Institut Teknologi Bandung (ITB), Universitas Indonesia, JAIST, NAISTIDSV test, all
mms-1b-allMetaIDSV test, all, zero-shotMMS fine-tuned, no LM (IDSV)Institut Teknologi Bandung (ITB), Universitas Indonesia, JAIST, NAISTIDSV test, all
whisper-smallOpenAIIDSV test, all, zero-shot
FT-whisper (IDSV)Institut Teknologi Bandung (ITB), Universitas Indonesia, JAIST, NAISTIDSV test, all groups14.85 WER
independent2MMS fine-tuned with LM (IDSV)Institut Teknologi Bandung (ITB), Universitas Indonesia, JAIST, NAISTIDSV test, all19.95 WER
independent3mms-1b-allMetaIDSV test, all, zero-shot29.3 WER
independent4MMS fine-tuned, no LM (IDSV)Institut Teknologi Bandung (ITB), Universitas Indonesia, JAIST, NAISTIDSV test, all29.66 WER
independent5whisper-smallOpenAIIDSV test, all, zero-shot30.87 WER
independent010203040
Score in CER, Table 3
Lower is better for this unit.
1FT-whisper (IDSV)Institut Teknologi Bandung (ITB), Universitas Indonesia, JAIST, NAISTIDSV test, all groupsMMS fine-tuned with LM (IDSV)Institut Teknologi Bandung (ITB), Universitas Indonesia, JAIST, NAISTIDSV test, allMMS fine-tuned, no LM (IDSV)Institut Teknologi Bandung (ITB), Universitas Indonesia, JAIST, NAISTIDSV test, allmms-1b-allMetaIDSV test, all, zero-shotwhisper-smallOpenAIIDSV test, all, zero-shot
FT-whisper (IDSV)Institut Teknologi Bandung (ITB), Universitas Indonesia, JAIST, NAISTIDSV test, all groups5.44 CER
independent2MMS fine-tuned with LM (IDSV)Institut Teknologi Bandung (ITB), Universitas Indonesia, JAIST, NAISTIDSV test, all6.45 CER
independent3MMS fine-tuned, no LM (IDSV)Institut Teknologi Bandung (ITB), Universitas Indonesia, JAIST, NAISTIDSV test, all7.86 CER
independent4mms-1b-allMetaIDSV test, all, zero-shot9.57 CER
independent5whisper-smallOpenAIIDSV test, all, zero-shot13.84 CER
independent051015
Where this comes from
Each bar is one model, with its most independent recorded score. Hatched bars are vendor-reported; the word beside each bar says who ran it. Settings and dates differ between rows; the Scores table below has every result.
Scores
| Model | Score | Who ran it | Setting | View or population | Date | Source |
|---|---|---|---|---|---|---|
| whisper-small | 30.87 WER | independent | IDSV test, all, zero-shot | Table 3 | Oct 2024 | arxiv.org |
| MMS fine-tuned, no LM (IDSV) | 29.66 WER | independent | IDSV test, all | Table 3 | Oct 2024 | arxiv.org |
| FT-whisper (IDSV) | 29.64 WER | independent | IDSV group SIC | Table 3, by speech variability | Oct 2024 | arxiv.org |
| mms-1b-all | 29.3 WER | independent | IDSV test, all, zero-shot | Table 3 | Oct 2024 | arxiv.org |
| MMS fine-tuned with LM (IDSV) | 19.95 WER | independent | IDSV test, all | Table 3 | Oct 2024 | arxiv.org |
| FT-whisper (IDSV) | 15.98 WER | independent | IDSV group SFM | Table 3, by speech variability | Oct 2024 | arxiv.org |
| FT-whisper (IDSV) | 14.85 WER | independent | IDSV test, all groups | Table 3 | Oct 2024 | arxiv.org |
| whisper-small | 13.84 CER | independent | IDSV test, all, zero-shot | Table 3 | Oct 2024 | arxiv.org |
| FT-whisper (IDSV) | 13.12 WER | independent | IDSV group RFM | Table 3, by speech variability | Oct 2024 | arxiv.org |
| FT-whisper (IDSV) | 13.04 WER | independent | IDSV group SIM | Table 3, by speech variability | Oct 2024 | arxiv.org |
| FT-whisper (IDSV) | 10.38 WER | independent | IDSV group RFC | Table 3, by speech variability | Oct 2024 | arxiv.org |
| mms-1b-all | 9.57 CER | independent | IDSV test, all, zero-shot | Table 3 | Oct 2024 | arxiv.org |
| MMS fine-tuned, no LM (IDSV) | 7.86 CER | independent | IDSV test, all | Table 3 | Oct 2024 | arxiv.org |
| MMS fine-tuned with LM (IDSV) | 6.45 CER | independent | IDSV test, all | Table 3 | Oct 2024 | arxiv.org |
| FT-whisper (IDSV) | 5.44 CER | independent | IDSV test, all groups | Table 3 | Oct 2024 | arxiv.org |
Where this comes from
Scores from different settings, harnesses or dates are not directly comparable; the row says which. Sorted by score, highest first, as a reading aid, not a ranking.
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "IDSV (Indonesian Speech Variability)", https://aurion.semai.id/benchmarks/idsv/, records of October 2026.