Phase 9: Advanced Topics & Capstone · 50 min · Python · OpenAI SDK · websockets

Native Audio Models: gpt-realtime-2, Nova Sonic, Moshi

Native audio models eliminate the text intermediary. The future of voice AI is speech-to-speech.

Hiring signal: Native audio model evaluation shows forward-thinking architecture awareness.

What you will learn

Evaluate native audio models: gpt-realtime-2, Nova 2 Sonic, Step-Audio R1.1, Moshi
Determine when native audio beats chained pipelines: latency, naturalness, prosody
Identify when chained pipelines win: control, cost, customization, provider flexibility
Understand the industry shift from chained pipelines to native audio

The Problem

Current voice agents chain ASR → LLM → TTS, adding 300-600ms of pipeline latency. Native audio models process speech directly — the model "hears" audio and "speaks" audio, eliminating the ASR and TTS steps entirely. This could cut latency to <200ms.

The Concept

Native Audio Model Architecture

Feature	Traditional (ASR+LLM+TTS)	Native Audio Model
Components	3 separate services	1 model
Latency	400-600ms	150-300ms (theoretical)
Audio understanding	Text only (loses tone, emotion)	Full audio (tone, emotion, accent)
Audio output	TTS synthesis (robotic)	Direct audio generation (natural)
Cost	3 providers	1 model (but expensive)
Maturity	Production-ready	Experimental

What's the key advantage of a native audio model over a traditional ASR+LLM+TTS pipeline?

Native audio models process audio directly — they hear tone, emotion, and accent (not just words), and generate natural speech (not TTS synthesis). This eliminates 2 pipeline steps (ASR + TTS), reducing latency from ~500ms to ~200ms, and preserves paralinguistic information that text-based pipelines lose

gpt-realtime-2 (OpenAI)

OpenAI's second-generation realtime model:

Feature	gpt-realtime-2	gpt-4o-realtime (v1)
Architecture	Native audio	ASR + GPT-4o + TTS
Latency	~200ms (target)	~400ms
Audio input	Direct audio tokens	Audio → text → LLM
Audio output	Direct audio generation	LLM → text → TTS
Emotion detection	Yes (from audio)	No (text only)
Function calling	Yes	Yes
Availability	Preview	Production

Nova Sonic (Amazon)

Amazon's native audio model for Alexa+:

Feature	Nova Sonic
Architecture	Native audio-to-audio
Latency	~250ms (target)
Integration	Alexa, AWS Bedrock
Strength	Wake-word-less conversations
Multilingual	Yes (planned)
Availability	Preview (AWS)

Moshi (Kyutai)

Open-source native audio model from Kyutai Research Lab:

Feature	Moshi
Architecture	Native audio-to-audio
Latency	~200ms (achieved)
Open source	Yes (Apache 2.0)
Model size	7B parameters
Full-duplex	Yes (simultaneous listen + speak)
Availability	Open weights, research

Full-Duplex Conversation

Traditional voice agents are half-duplex: listen → think → speak → listen. Native audio models like Moshi can be full-duplex — they listen and speak simultaneously, like human conversation.

class NativeAudioModel:
    """Simulates a native audio model."""

    async def process_audio(self, input_audio):
        """Process audio directly — no ASR/TTS."""
        # Model "hears" audio and "speaks" audio
        # No text intermediate representation
        start = time.time()

        # Simulate native audio processing
        await asyncio.sleep(0.15)  # ~150ms latency

        # Generate audio response directly
        output_audio = self._generate_audio_response(input_audio)

        latency = (time.time() - start) * 1000
        return {
            "audio": output_audio,
            "latency_ms": latency,
            "emotion_detected": "neutral",
            "model": "native_audio",
        }

Comparison: Latency Breakdown

Component	Traditional	Native Audio
Network in	30ms	30ms
VAD	10ms	10ms
Endpointing	300ms	100ms (faster — audio-level detection)
ASR	150ms	0ms (eliminated)
LLM	200ms	150ms (audio model)
TTS	120ms	0ms (eliminated)
Network out	30ms	30ms
Total	840ms	320ms

Challenges

Challenge	Description	Status
Hallucination	Audio models can generate sounds, not just speech	Research
Latency in practice	Theoretical 200ms vs actual 300-400ms	Improving
Cost	Audio tokens are 10-100x more expensive than text	High
Function calling	Harder with audio-only models	Being solved
Context window	Audio uses more tokens than text	Limiting
Fine-tuning	Can't easily fine-tune audio models	Research
Ecosystem	No RAG, no tools, no guardrails yet	Early

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Latency