Phase 7: Speech Synthesis & Applications · ~60 minutes · Python

Audio Evaluation — WER, MOS, UTMOS, MMAU, FAD, and the Open Leaderboards

You cannot ship what you cannot measure. This lesson names the 2026 metrics for every audio task: ASR (WER, CER, RTFx), TTS (MOS, UTMOS, SECS, WER-on-ASR-round-trip), audio-language (MMAU, LongAudioBench), music (FAD, CLAP), and speaker (EER). Plus the leaderboards where you compare.

Hiring signal: Understanding of audio evaluation — wer, mos, utmos, mmau, fad, and the open leaderboards internals

What you will learn

Implement audio evaluation — wer, mos, utmos, mmau, fad, and the open leaderboards from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Learn Languages: Python Prerequisites: Phase 6 · 04, 06, 07, 09, 10; Phase 2 · 09 (Model Evaluation) Time: ~60 minutes

The Problem

Every audio task has multiple metrics, each measuring a different axis. Using the wrong metric is how you ship a model that looks great on your dashboard and terribly in production. The 2026 canonical list:

Task	Primary	Secondary
ASR	WER	CER · RTFx · first-token latency
TTS	MOS / UTMOS	SECS · WER-on-ASR-round-trip · CER · TTFA
Voice cloning	SECS (ECAPA cosine)	MOS · CER
Speaker verification	EER	minDCF · FAR / FRR at operating point
Diarization	DER	JER · speaker confusion
Audio classification	top-1 · mAP	macro F1 · per-class recall
Music generation	FAD	CLAP · listening panel MOS
Audio language model	MMAU-Pro	LongAudioBench · AudioCaps FENSE
Streaming S2S	latency P50/P95	WER · MOS

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build It, Use It, Pitfalls, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.