Phase 2: Speech-to-Text (ASR) · 55 min · Python · Deepgram SDK · websockets

Deepgram Nova-3 Integration

Deepgram is the latency king. If your voice agent needs speed, this is your ASR.

Hiring signal: Deepgram integration is the most common ASR coding exercise in voice AI interviews.

What you will learn

Integrate Deepgram Nova-3 via WebSocket for streaming ASR
Handle Deepgram events: Results, Metadata, SpeechFinal
Configure custom vocabulary and jargon hints
Implement diarization for multi-speaker calls

The Problem

Deepgram Nova-3 is the most popular streaming ASR for voice agents — 150ms TTFT, 5% WER, $0.0077/min. But integrating it correctly requires understanding its WebSocket API, event types, configuration options, and error handling. Get it wrong and you'll have garbled transcripts, missed utterances, or connection drops.

The Concept

Deepgram Streaming Architecture

Key Configuration Options

Option	Default	Description	When to Change
`model`	`nova-2`	ASR model	Use `nova-3` for best accuracy
`encoding`	`linear16`	Audio encoding	`mulaw` for telephony
`sample_rate`	`16000`	Audio sample rate	`8000` for telephony
`channels`	`1`	Audio channels	Always 1 for voice
`interim_results`	`true`	Send partials	`false` for finals only
`endpointing`	`300`	Silence detection (ms)	Lower for faster response
`vad_events`	`false`	VAD events	`true` for custom endpointing
`diarize`	`false`	Speaker diarization	`true` for multi-speaker
`smart_format`	`true`	Punctuation, formatting	Keep on for readability
`filler_words`	`false`	Include "um", "uh"	`true` for analytics
`keywords`	`[]`	Custom vocabulary hints	Add domain terms

Event Types

Event	Description	Action
`Results`	Partial or final transcript	Update UI (partial) or trigger LLM (final)
`Metadata`	Request metadata, request_id	Log for debugging
`SpeechStarted`	Speech detected	Start recording, update UI
`UtteranceEnd`	Silence exceeded threshold	Alternative to SpeechFinal
`Error`	Connection or processing error	Reconnect, log

You're building a voice agent for a healthcare company. Callers say drug names like "metformin" and "atorvastatin" that ASR misrecognizes. What Deepgram feature do you use?

diarize

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Streaming