The Concept
Word Error Rate (WER)
WER is the standard ASR quality metric. It measures the edit distance between the ASR output and the ground truth:
WER = (S + D + I) / N
S = Substitutions (wrong words)
D = Deletions (missing words)
I = Insertions (extra words)
N = Total words in reference
def calculate_wer(reference, hypothesis):
"""Calculate Word Error Rate."""
ref_words = reference.lower().split()
hyp_words = hypothesis.lower().split()
# Levenshtein distance at word level
d = [[0] * (len(hyp_words) + 1) for _ in range(len(ref_words) + 1)]
for i in range(len(ref_words) + 1):
d[i][0] = i
for j in range(len(hyp_words) + 1):
d[0][j] = j
for i in range(1, len(ref_words) + 1):
for j in range(1, len(hyp_words) + 1):
if ref_words[i-1] == hyp_words[j-1]:
d[i][j] = d[i-1][j-1]
else:
d[i][j] = min(
d[i-1][j] + 1, # Deletion
d[i][j-1] + 1, # Insertion
d[i-1][j-1] + 1 # Substitution
)
# Backtrack to count S, D, I
i, j = len(ref_words), len(hyp_words)
substitutions = deletions = insertions = 0
while i > 0 or j > 0:
if i > 0 and j > 0 and ref_words[i-1] == hyp_words[j-1]:
i -= 1
j -= 1
elif i > 0 and d[i][j] == d[i-1][j] + 1:
deletions += 1
i -= 1
elif j > 0 and d[i][j] == d[i][j-1] + 1:
insertions += 1
j -= 1
else:
substitutions += 1
i -= 1
j -= 1
wer = (substitutions + deletions + insertions) / len(ref_words)
return {
"wer": wer,
"substitutions": substitutions,
"deletions": deletions,
"insertions": insertions,
"ref_words": len(ref_words),
}
An ASR system transcribes "I want to book a flight" as "I want too book a flight". What is the WER?
WER = (S + D + I) / N = (1 + 0 + 0) / 7 = 14.3%. The substitution of "to" → "too" counts as one error even though they're homophones. WER is a word-level metric — it doesn't understand that "to" and "too" sound identical. This is a known limitation: WER overcounts errors for homophones. Some evaluators use 'semantic WER' or 'WER with homophone normalization' to address this.
The Evaluation Pipeline
Character Error Rate (CER)
CER is the same concept but at character level — useful for languages without clear word boundaries (Chinese, Japanese):
def calculate_cer(reference, hypothesis):
"""Calculate Character Error Rate."""
ref_chars = list(reference.lower())
hyp_chars = list(hypothesis.lower())
# Same Levenshtein distance but at character level
# ...
Time to First Token (TTFT)
TTFT measures when the first partial transcript arrives after audio starts streaming:
import time
async def measure_ttft(asr_provider, audio_stream):
"""Measure Time to First Token for an ASR provider."""
start = time.time()
first_token_time = None
async for event in asr_provider.transcribe(audio_stream):
if event["type"] == "partial" and event["text"]:
first_token_time = time.time() - start
break
return first_token_time
Latency Distribution: P50, P95, P99
Single measurements are misleading. You need the distribution:
| Percentile | Meaning | Target for Voice Agent |
|---|
| P50 | 50% of requests faster than this | <200ms |
| P95 | 95% of requests faster than this | <500ms |
| P99 | 99% of requests faster than this | <800ms |
def calculate_percentiles(latencies):
"""Calculate P50, P95, P99."""
sorted_latencies = sorted(latencies)
n = len(sorted_latencies)
return {
"p50": sorted_latencies[int(n * 0.50)],
"p95": sorted_latencies[int(n * 0.95)],
"p99": sorted_latencies[int(n * 0.99)],
"min": sorted_latencies[0],
"max": sorted_latencies[-1],
"mean": sum(sorted_latencies) / n,
}
Testing on Diverse Accents
| Accent | Deepgram WER | AssemblyAI WER | Whisper WER |
|---|
| Standard American | 5% | 7% | 3% |
| Indian English | 12% | 8% | 6% |
| Scottish | 15% | 10% | 8% |
| Southern US | 10% | 8% | 6% |
| Australian | 8% | 7% | 5% |
| Chinese English | 18% | 12% | 10% |
Provider Selection Framework
def select_asr_provider(use_case, requirements):
"""Select ASR provider based on requirements."""
scores = {}
for provider in PROVIDERS:
score = 0
if requirements.get("latency") == "low":
score += provider.ttft_score
if requirements.get("accuracy") == "high":
score += provider.wer_score
if requirements.get("accents"):
score += provider.accent_score
if requirements.get("cost") == "low":
score += provider.cost_score
if requirements.get("streaming"):
score += provider.streaming_score
scores[provider.name] = score
return max(scores, key=scores.get)