The Concept
TTS Quality Metrics
| Metric | What It Measures | Scale | Target |
|---|
| MOS | Mean Opinion Score (subjective quality) | 1-5 | >4.0 |
| CMOS | Comparative MOS (A vs B) | -3 to +3 | >0 |
| MUSHRA | Multiple stimuli comparison | 0-100 | >80 |
| TTFB | Time to First Byte | ms | <200ms |
| RTF | Real-Time Factor (processing/audio) | ratio | <1.0 |
| WER-TTS | ASR on TTS output (intelligibility) | % | <5% |
MOS (Mean Opinion Score)
MOS is the standard TTS quality metric, rated by human listeners:
| Score | Quality | Description |
|---|
| 5 | Excellent | Imperceptible from human speech |
| 4 | Good | Perceptible but not annoying |
| 3 | Fair | Slightly annoying artifacts |
| 2 | Poor | Very annoying artifacts |
| 1 | Bad | Unintelligible |
def calculate_mos(ratings):
"""Calculate MOS from human ratings (1-5 scale)."""
return sum(ratings) / len(ratings)
# Example: 10 listeners rated ElevenLabs 4.5, Cartesia 4.2
elevenlabs_mos = calculate_mos([5, 4, 5, 4, 5, 4, 4, 5, 4, 4]) # 4.4
cartesia_mos = calculate_mos([4, 4, 5, 4, 4, 4, 4, 5, 4, 4]) # 4.2
TTFB Measurement
import time
async def measure_ttfb(tts_provider, text):
"""Measure TTS Time to First Byte."""
start = time.time()
async for chunk in tts_provider.stream(text):
ttfb = (time.time() - start) * 1000
return ttfb
return None
Intelligibility Testing (WER-TTS)
A clever way to measure TTS quality: run ASR on TTS output and measure WER:
async def measure_intelligibility(tts_provider, text, asr_provider):
"""Measure TTS intelligibility via ASR round-trip."""
# Generate TTS audio
audio = await tts_provider.synthesize(text)
# Run ASR on the TTS audio
transcript = await asr_provider.transcribe(audio)
# Calculate WER between original text and ASR output
wer = calculate_wer(text, transcript)
return wer # Lower = more intelligible
Why might a TTS system have high MOS but low intelligibility (high WER-TTS)?
MOS is unreliable
The TTS Evaluation Pipeline
Test Content Categories
| Category | Example | Why Test |
|---|
| Simple sentences | "Hello, how can I help you?" | Baseline quality |
| Numbers | "Your balance is $1,247.89" | Number pronunciation |
| Proper nouns | "Flight UA 2487 to Tokyo" | Name/term pronunciation |
| Long sentences | "I can help you book a flight..." | Breath, pacing |
| Questions | "Would you prefer morning or evening?" | Intonation |
| Commands | "Please say your account number." | Clarity, authority |
| Domain terms | "Your ETF dividend is compound interest" | Jargon pronunciation |
| Multi-language | "Your reservation for restaurante is confirmed" | Code-switching |
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.