Audio Language Models for Speech Understanding
Language models that take audio as input to transcribe, answer questions about and converse over speech, sound and music.
Tier Grades
Tier A: Frontier AI labs
feasible
Mistral and Moonshot built these with 106 and 40 authors and 13 million hours of audio.
Tier B: Indonesian AI labs
stretch
Continued pre-training of an LLM on audio is possible only at far smaller scale than Kimi-Audio's 13 million hours.
Tier C: Institution units such as ITB
not feasible
A unit cannot assemble 13 million hours of audio or a team of 40 or more, though it can evaluate released models.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
Open models such as Voxtral under Apache 2.0 can be used directly for long meeting audio up to 40 minutes.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeNot in the sources read: the Voxtral and Kimi-Audio abstracts give no parameter count or training compute; Kimi-Audio is initialised from a pre-trained LLM.
- DataKimi-Audio: more than 13 million hours of speech, sound and music for pre-training; Voxtral: 32K context handles audio up to 40 minutes.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 106 authors on Voxtral; 40 listed on Kimi-Audio.
- Time horizonMonths to a year.
- Shelf-life signalsGeneral multimodal models matching dedicated recognisers on transcription accuracy and cost
- Pipeline roleTier A trains audio language models; Tier B and Tier C evaluate them on Indonesian speech; Tier D uses them for meeting minutes and call analysis pilots.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Linked models
Qwen3-ASRspeech recognitionexemplar
Canary-Qwen-2.5Bspeech recognitionexemplar
MERaLiON-2multimodal LLMexemplar
Ultravoxmultimodal LLMexemplar
Voxtral Smallspeech recognitionexemplar
Kimi-Audiospeech recognitionexemplar
Moshi (Moshiko)multimodal LLMexemplarQwen2-Audio-7B-Instructmultimodal LLMexemplar
granite-speech-4.1-2bspeech recognitionproduced byQwen3-ASR-0.6Bspeech recognitionproduced byQwen3-ASR-1.7Bspeech recognitionproduced by
Step-Audio-R1.1multimodal LLMproduced byFun-ASR-Nano (Fun-ASR-Nano-2512)speech recognitionproduced by
SeaLLMs-Audio-7Bmultimodal LLMproduced byAudio Flamingo 3multimodal LLMproduced byVoxtral Mini 3B 2507speech recognitionproduced byMERaLiON-2-3Bmultimodal LLMproduced byMERaLiON-2-10Bmultimodal LLMproduced byStep-Audiomultimodal LLMproduced byStep-Audio-Chatmultimodal LLMproduced byUltravox v0.5 Llama 3.2 1Bmultimodal LLMproduced byUltravox v0.5 Llama 3.1 8Bmultimodal LLMproduced bygranite-speech-3.3-8bspeech recognitionproduced by
Phi-4-multimodal-instructmultimodal LLMproduced byLinks
R&D TopicAudio Language Models for Speech Understanding
ModelsQwen3-ASRAlibaba (Qwen), speech recognitionVoxtral SmallMistral AI, speech recognitionKimi-AudioMoonshot AI, speech recognitionCanary-Qwen-2.5BNVIDIA, speech recognition+23 more
Qwen3-ASRAlibaba (Qwen), speech recognitionVoxtral SmallMistral AI, speech recognitionKimi-AudioMoonshot AI, speech recognitionCanary-Qwen-2.5BNVIDIA, speech recognition+23 moreUse Casesnone recorded
Who Runs Itnone recorded
Models 27
MERaLiON-2-10BA*STAR (Institute for Infocomm Research, I2R), multimodal LLMproduced byMERaLiON-2-3BA*STAR (Institute for Infocomm Research, I2R), multimodal LLMproduced byMERaLiON-2A*STAR (Institute for Infocomm Research, I2R), multimodal LLMexemplarQwen3-ASRAlibaba (Qwen), speech recognitionexemplarCanary-Qwen-2.5BNVIDIA, speech recognitionexemplargranite-speech-4.1-2bIBM, speech recognitionproduced byStep-AudioStepFun, multimodal LLMproduced byUltravoxFixie.ai, multimodal LLMexemplarFun-ASR-Nano (Fun-ASR-Nano-2512)Alibaba (Qwen), speech recognitionproduced bygranite-speech-3.3-8bIBM, speech recognitionproduced byMoshi (Moshiko)Kyutai, multimodal LLMexemplarPhi-4-multimodal-instructMicrosoft AI, multimodal LLMproduced by+15
Qwen3-ASR-0.6BAlibaba (Qwen), speech recognitionproduced byQwen3-ASR-1.7BAlibaba (Qwen), speech recognitionproduced byQwen3-ASR-Flash-1208Alibaba (Qwen), speech recognitionproduced bySeaLLMs-Audio-7BAlibaba DAMO, multimodal LLMproduced byStep-Audio-ChatStepFun, multimodal LLMproduced byStep-Audio-R1.1StepFun, multimodal LLMproduced byUltravox v0.5 Llama 3.1 8BFixie.ai, multimodal LLMproduced byUltravox v0.5 Llama 3.2 1BFixie.ai, multimodal LLMproduced byAudio Flamingo 3NVIDIA, multimodal LLMproduced by
gpt-4o-transcribeOpenAI, speech recognitionproduced byKimi-AudioMoonshot AI, speech recognitionexemplarQwen2-Audio-7B-InstructAlibaba (Qwen), multimodal LLMexemplar
Shuka v1Sarvam AI, multimodal LLMproduced byVoxtral Mini 3B 2507Mistral AI, speech recognitionproduced byVoxtral SmallMistral AI, speech recognitionexemplarPapers
- AudioLM: a Language Modeling Approach to Audio GenerationZalán Borsos et al., arXiv preprint, 2022 founding
- Qwen2-Audio Technical ReportYunfei Chu et al., arXiv preprint, 2024 state of the art
- Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue AbilitiesZhifeng Kong et al., ICML 2024 (per arXiv comment), 2024 state of the art
- SALMONN: Towards Generic Hearing Abilities for Large Language ModelsChangli Tang et al., arXiv preprint, 2023 state of the art
- Recent Advances in Speech Language Models: A SurveyWenqian Cui et al., ACL 2025 (per arXiv comment), 2024 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- arxiv.org/abs/2507.13264 https://arxiv.org/abs/2507.13264
- arxiv.org/abs/2504.18425 https://arxiv.org/abs/2504.18425
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Audio Language Models for Speech Understanding", https://aurion.semai.id/topics/audio-language-models/, records of October 2026.