The Concept
Word Error Rate (WER)
WER is the standard ASR quality metric. It measures the edit distance between the ASR output and the ground truth:
WER = (S + D + I) / N
S = Substitutions (wrong words)
D = Deletions (missing words)
I = Insertions (extra words)
N = Total words in reference
def calculate_wer(reference, hypothesis):
"""Calculate Word Error Rate."""
ref_words = reference.lower().split()
hyp_words = hypothesis.lower().split()
# Levenshtein distance at word level
d = [[0] * (len(hyp_words) + 1) for _ in range(len(ref_words) + 1)]
for i in range(len(ref_words) + 1):
d[i][0] = i
for j in range(len(hyp_words) + 1):
d[0][j] = j
for i in range(1, len(ref_words) + 1):
for j in range(1, len(hyp_words) + 1):
if ref_words[i-1] == hyp_words[j-1]:
d[i][j] = d[i-1][j-1]
else:
d[i][j] = min(
d[i-1][j] + 1, # Deletion
d[i][j-1] + 1, # Insertion
d[i-1][j-1] + 1 # Substitution
)
# Backtrack to count S, D, I
i, j = len(ref_words), len(hyp_words)
substitutions = deletions = insertions = 0
while i > 0 or j > 0:
if i > 0 and j > 0 and ref_words[i-1] == hyp_words[j-1]:
i -= 1
j -= 1
elif i > 0 and d[i][j] == d[i-1][j] + 1:
deletions += 1
i -= 1
elif j > 0 and d[i][j] == d[i][j-1] + 1:
insertions += 1
j -= 1
else:
substitutions += 1
i -= 1
j -= 1
wer = (substitutions + deletions + insertions) / len(ref_words)
return {
"wer": wer,
"substitutions": substitutions,
"deletions": deletions,
"insertions": insertions,
"ref_words": len(ref_words),
}
An ASR system transcribes "I want to book a flight" as "I want too book a flight". What is the WER?
50% — half the words are wrong
The Evaluation Pipeline
Character Error Rate (CER)
CER is the same concept but at character level — useful for languages without clear word boundaries (Chinese, Japanese):
def calculate_cer(reference, hypothesis):
"""Calculate Character Error Rate."""
ref_chars = list(reference.lower())
hyp_chars = list(hypothesis.lower())
# Same Levenshtein distance but at character level
# ...
Time to First Token (TTFT)
TTFT measures when the first partial transcript arrives after audio starts streaming:
import time
async def measure_ttft(asr_provider, audio_stream):
"""Measure Time to First Token for an ASR provider."""
start = time.time()
first_token_time = None
async for event in asr_provider.transcribe(audio_stream):
if event["type"] == "partial" and event["text"]:
first_token_time = time.time() - start
break
return first_token_time
Latency Distribution: P50, P95, P99
Single measurements are misleading. You need the distribution:
| Percentile | Meaning | Target for Voice Agent |
|---|
| P50 | 50% of requests faster than this | <200ms |
| P95 | 95% of requests faster than this | <500ms |
| P99 | 99% of requests faster than this | <800ms |
def calculate_percentiles(latencies):
"""Calculate P50, P95, P99."""
sorted_latencies = sorted(latencies)
n = len(sorted_latencies)
return {
"p50": sorted_latencies[int(n * 0.50)],
"p95": sorted_latencies[int(n * 0.95)],
"p99": sorted_latencies[int(n * 0.99)],
"min": sorted_latencies[0],
"max": sorted_latencies[-1],
"mean": sum(sorted_latencies) / n,
}
Testing on Diverse Accents
| Accent | Deepgram WER | AssemblyAI WER | Whisper WER |
|---|
| Standard American | 5% | 7% | 3% |
| Indian English | 12% | 8% | 6% |
| Scottish | 15% | 10% | 8% |
| Southern US | 10% | 8% | 6% |
| Australian | 8% | 7% | 5% |
| Chinese English | 18% | 12% | 10% |
Provider Selection Framework
def select_asr_provider(use_case, requirements):
"""Select ASR provider based on requirements."""
scores = {}
for provider in PROVIDERS:
score = 0
if requirements.get("latency") == "low":
score += provider.ttft_score
if requirements.get("accuracy") == "high":
score += provider.wer_score
if requirements.get("accents"):
score += provider.accent_score
if requirements.get("cost") == "low":
score += provider.cost_score
if requirements.get("streaming"):
score += provider.streaming_score
scores[provider.name] = score
return max(scores, key=scores.get)