Phase 2: Speech-to-Text (ASR) · 50 min · Python · AssemblyAI SDK · websockets

AssemblyAI Universal-2 Integration

When accents matter, AssemblyAI Universal-2 is the accuracy champion.

Hiring signal: AssemblyAI integration and accent handling shows provider selection reasoning beyond default choices.

What you will learn

Integrate AssemblyAI Universal-2 for accented English accuracy
Use the Voice Agent API with built-in turn detection
Handle accents, background noise, code-switching, and disfluencies
Compare AssemblyAI vs Deepgram for accented English use cases

The Problem

Deepgram Nova-3 is the default choice for streaming ASR, but it struggles with:

Heavy accents (Indian, Scottish, Southern US)
Code-switching (mixing languages mid-sentence)
Background noise in mobile environments
Multi-speaker diarization

AssemblyAI Universal-2 is specifically trained to handle these edge cases. Knowing when to switch providers is what makes you a senior voice AI engineer.

The Concept

AssemblyAI vs Deepgram

Feature	Deepgram Nova-3	AssemblyAI Universal-2
WER (standard English)	~5%	~7%
WER (accented English)	~12%	~8%
TTFT	~150ms	~300ms
Diarization	Good	Excellent
Price	$0.0077/min	$0.012/min
Streaming	WebSocket	WebSocket
Batch	Yes	Yes
Post-call analytics	Basic	LeMUR (LLM-powered)

When to Choose AssemblyAI

Accented English: Callers from India, Philippines, UK, Australia
Multi-speaker: Conference calls, agent + customer
Post-call analytics: Need summarization, sentiment, topics
Code-switching: Callers mixing languages
Noisy environments: Mobile calls, cars, public spaces

Dual-Provider Routing Flow

AssemblyAI Streaming API

import assemblyai as aai

aai.settings.api_key = os.getenv("ASSEMBLYAI_API_KEY")

# Streaming transcription
transcriber = aai.RealtimeTranscriber(
    sample_rate=16000,
    on_data=on_data,
    on_error=on_error,
    on_open=on_open,
    on_close=on_close,
)
transcriber.connect()

Event Handling

def on_data(data: aai.RealtimeTranscript):
    if data.text:
        if data.end_of_turn:
            # Final transcript — trigger LLM
            print(f"Final: {data.text}")
        else:
            # Partial — update UI only
            print(f"Partial: {data.text}")

Post-Call Analytics with LeMUR

AssemblyAI's LeMUR framework provides LLM-powered post-call analysis:

# After call ends, run batch transcription + analysis
transcript = aai.Transcriber().transcribe("call_recording.wav")

# Summarization
summary = aai.Lemur().summarize(
    transcript, context="customer support call about flight booking"
)

# Sentiment analysis
sentiment = aai.Lemur().ask_question(
    transcript, question="What was the customer's emotional state?"
)

# Action items
actions = aai.Lemur().ask_question(
    transcript, question="What follow-up actions are needed?"
)

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.