Phase 2: Speech-to-Text (ASR) · 50 min · Python · faster-whisper · OpenAI SDK

OpenAI Whisper and Self-Hosted ASR

Whisper covers 99 languages. Self-hosted gives you privacy and cost control at scale.

Hiring signal: Self-hosted ASR knowledge shows cost optimization and privacy awareness for enterprise deployments.

What you will learn

Use OpenAI Whisper for multilingual batch transcription
Run self-hosted Whisper with faster-whisper for privacy-sensitive deployments
Evaluate ElevenLabs Scribe for post-call analysis
Compare hosted vs self-hosted ASR: cost, latency, privacy, maintenance

The Problem

Hosted ASR providers charge per-minute fees that add up at scale. For a voice agent handling 100K calls/month at 3 minutes each, ASR costs $2,310/month with Deepgram. Self-hosted Whisper eliminates per-minute fees but requires GPU infrastructure and adds latency.

The question: when does self-hosting make sense, and how do you integrate it?

The Concept

Whisper Architecture (Batch Only)

Whisper is an encoder-decoder Transformer trained on 680,000 hours of multilingual data:

Encoder: Processes mel-spectrogram of full audio
Decoder: Auto-regressively generates text tokens with cross-attention
Cannot stream: Needs full audio before decoding starts
Languages: 99 languages with automatic detection
Accuracy: Best-in-class for batch transcription

faster-whisper: Making Whisper Practical

faster-whisper is a CTranslate2-based implementation that's 4x faster than OpenAI's Whisper with lower memory:

Model	VRAM	Speed (16kHz, 1min audio)	WER
tiny	~1GB	~2s	~12%
base	~1GB	~3s	~9%
small	~2GB	~5s	~7%
medium	~5GB	~10s	~5%
large-v3	~10GB	~20s	~3%

Why can't Whisper do streaming ASR?

It doesn't support WebSocket

Choosing Hosted vs Self-Hosted ASR

Pseudo-Streaming with faster-whisper

While Whisper can't truly stream, you can chunk audio and process segments:

from faster_whisper import WhisperModel

model = WhisperModel("base", device="cuda", compute_type="int8")

# Pseudo-streaming: process 2-second chunks
def pseudo_stream(audio_stream, chunk_duration=2.0):
    buffer = []
    for frame in audio_stream:
        buffer.append(frame)
        if len(buffer) >= int(16000 * chunk_duration):
            chunk = np.concatenate(buffer)
            segments, _ = model.transcribe(chunk, language="en")
            for seg in segments:
                yield seg.text
            buffer = []

This adds ~2-5s latency per chunk but works for non-real-time use cases.

Self-Hosted vs Hosted: Cost Analysis

Volume	Deepgram ($0.0077/min)	Self-hosted (GPU $0.80/hr)	Break-even
1K calls/min/month	$23	$585 (1 GPU)	Not worth it
10K calls/min/month	$231	$585 (1 GPU)	Not worth it
50K calls/min/month	$1,155	$585 (1 GPU)	Self-host wins
100K calls/min/month	$2,310	$1,170 (2 GPUs)	Self-host wins
500K calls/min/month	$11,550	$4,680 (8 GPUs)	Self-host wins

Break-even: ~30K minutes/month (one GPU can handle ~30K min/month with batching)

When to Self-Host

Factor	Self-Host	Hosted
Volume > 30K min/month	✅
Privacy/ compliance (HIPAA, GDPR)	✅
Need offline/air-gapped	✅
Low latency (streaming)		✅
Quick setup		✅
Multilingual (99 languages)	✅ (Whisper)	✅ (Deepgram)
Custom model fine-tuning	✅
Budget < $500/month		✅

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.