Phase 2: Speech-to-Text (ASR) · 55 min · Python · Deepgram SDK · websockets

Deepgram Nova-3 Integration

Deepgram is the latency king. If your voice agent needs speed, this is your ASR.

What you will learn

Integrate Deepgram Nova-3 via WebSocket for streaming ASR
Handle Deepgram events: Results, Metadata, SpeechFinal
Configure custom vocabulary and jargon hints
Implement diarization for multi-speaker calls

The Problem

Deepgram Nova-3 is the most popular streaming ASR for voice agents — 150ms TTFT, 5% WER, $0.0077/min. But integrating it correctly requires understanding its WebSocket API, event types, configuration options, and error handling. Get it wrong and you'll have garbled transcripts, missed utterances, or connection drops.

The Concept

Deepgram Streaming Architecture

Key Configuration Options

Option	Default	Description	When to Change
`model`	`nova-2`	ASR model	Use `nova-3` for best accuracy
`encoding`	`linear16`	Audio encoding	`mulaw` for telephony
`sample_rate`	`16000`	Audio sample rate	`8000` for telephony
`channels`	`1`	Audio channels	Always 1 for voice
`interim_results`	`true`	Send partials	`false` for finals only
`endpointing`	`300`	Silence detection (ms)	Lower for faster response
`vad_events`	`false`	VAD events	`true` for custom endpointing
`diarize`	`false`	Speaker diarization	`true` for multi-speaker
`smart_format`	`true`	Punctuation, formatting	Keep on for readability
`filler_words`	`false`	Include "um", "uh"	`true` for analytics
`keywords`	`[]`	Custom vocabulary hints	Add domain terms

Event Types

Event	Description	Action
`Results`	Partial or final transcript	Update UI (partial) or trigger LLM (final)
`Metadata`	Request metadata, request_id	Log for debugging
`SpeechStarted`	Speech detected	Start recording, update UI
`UtteranceEnd`	Silence exceeded threshold	Alternative to SpeechFinal
`Error`	Connection or processing error	Reconnect, log

You're building a voice agent for a healthcare company. Callers say drug names like "metformin" and "atorvastatin" that ASR misrecognizes. What Deepgram feature do you use?

The keywords parameter lets you pass domain-specific terms that Deepgram's model should bias toward. For healthcare: keywords=["metformin", "atorvastatin", "lisinopril", "amoxicillin"]. For finance: keywords=["portfolio", "dividend", "ETF"]. This is the single most impactful configuration for domain-specific ASR accuracy — it can reduce WER by 50%+ for specialized vocabulary.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.

Key terms in this lesson

Streaming