The Concept
Native Audio Model Architecture
| Feature | Traditional (ASR+LLM+TTS) | Native Audio Model |
|---|
| Components | 3 separate services | 1 model |
| Latency | 400-600ms | 150-300ms (theoretical) |
| Audio understanding | Text only (loses tone, emotion) | Full audio (tone, emotion, accent) |
| Audio output | TTS synthesis (robotic) | Direct audio generation (natural) |
| Cost | 3 providers | 1 model (but expensive) |
| Maturity | Production-ready | Experimental |
What's the key advantage of a native audio model over a traditional ASR+LLM+TTS pipeline?
The key advantage is twofold: (1) Latency — eliminating ASR and TTS from the pipeline removes ~200-300ms of processing. The model directly maps input audio to output audio, potentially achieving <200ms glass-to-glass latency. (2) Audio fidelity — traditional pipelines convert speech to text (losing tone, emotion, accent, hesitation) then text back to speech (TTS sounds synthetic). Native audio models preserve the full audio signal — they can detect sarcasm, urgency, or confusion from tone of voice, and generate responses with natural intonation, pauses, and emotional expression. The disadvantages: native audio models are experimental, expensive (more compute than text models), and lack the ecosystem (tools, function calling, RAG) that text LLMs have. They're the future but not yet production-ready for most use cases.
gpt-realtime-2 (OpenAI)
OpenAI's second-generation realtime model:
| Feature | gpt-realtime-2 | gpt-4o-realtime (v1) |
|---|
| Architecture | Native audio | ASR + GPT-4o + TTS |
| Latency | ~200ms (target) | ~400ms |
| Audio input | Direct audio tokens | Audio → text → LLM |
| Audio output | Direct audio generation | LLM → text → TTS |
| Emotion detection | Yes (from audio) | No (text only) |
| Function calling | Yes | Yes |
| Availability | Preview | Production |
Nova Sonic (Amazon)
Amazon's native audio model for Alexa+:
| Feature | Nova Sonic |
|---|
| Architecture | Native audio-to-audio |
| Latency | ~250ms (target) |
| Integration | Alexa, AWS Bedrock |
| Strength | Wake-word-less conversations |
| Multilingual | Yes (planned) |
| Availability | Preview (AWS) |
Moshi (Kyutai)
Open-source native audio model from Kyutai Research Lab:
| Feature | Moshi |
|---|
| Architecture | Native audio-to-audio |
| Latency | ~200ms (achieved) |
| Open source | Yes (Apache 2.0) |
| Model size | 7B parameters |
| Full-duplex | Yes (simultaneous listen + speak) |
| Availability | Open weights, research |
Full-Duplex Conversation
Traditional voice agents are half-duplex: listen → think → speak → listen. Native audio models like Moshi can be full-duplex — they listen and speak simultaneously, like human conversation.
class NativeAudioModel:
"""Simulates a native audio model."""
async def process_audio(self, input_audio):
"""Process audio directly — no ASR/TTS."""
# Model "hears" audio and "speaks" audio
# No text intermediate representation
start = time.time()
# Simulate native audio processing
await asyncio.sleep(0.15) # ~150ms latency
# Generate audio response directly
output_audio = self._generate_audio_response(input_audio)
latency = (time.time() - start) * 1000
return {
"audio": output_audio,
"latency_ms": latency,
"emotion_detected": "neutral",
"model": "native_audio",
}
Comparison: Latency Breakdown
| Component | Traditional | Native Audio |
|---|
| Network in | 30ms | 30ms |
| VAD | 10ms | 10ms |
| Endpointing | 300ms | 100ms (faster — audio-level detection) |
| ASR | 150ms | 0ms (eliminated) |
| LLM | 200ms | 150ms (audio model) |
| TTS | 120ms | 0ms (eliminated) |
| Network out | 30ms | 30ms |
| Total | 840ms | 320ms |
Challenges
| Challenge | Description | Status |
|---|
| Hallucination | Audio models can generate sounds, not just speech | Research |
| Latency in practice | Theoretical 200ms vs actual 300-400ms | Improving |
| Cost | Audio tokens are 10-100x more expensive than text | High |
| Function calling | Harder with audio-only models | Being solved |
| Context window | Audio uses more tokens than text | Limiting |
| Fine-tuning | Can't easily fine-tune audio models | Research |
| Ecosystem | No RAG, no tools, no guardrails yet | Early |
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.