The Problem
Production voice agents fail silently. A provider degrades, latency creeps up, an attacker probes your endpoints, or PII leaks through transcripts. Without monitoring and alerting, you find out from Twitter complaints or compliance fines. Security isn't optional — voice agents handle sensitive data (PII, payment info, health records) and are subject to regulations.
The Concept
Monitoring Stack
| Layer | Tool | What It Monitors |
|---|
| Metrics | Prometheus + Grafana | Latency, error rate, call volume, CPU |
| Logs | ELK (Elasticsearch, Logstash, Kibana) | Structured JSON logs per call |
| Errors | Sentry | Exceptions, stack traces, error trends |
| Alerts | AlertManager + PagerDuty | Critical issues → on-call |
| Tracing | Jaeger/Tempo | Request flow across services |
| Uptime | UptimeRobot/Pingdom | External availability checks |
What's the difference between a metric and a log?
Metrics are aggregated numeric data (e.g., "P95 latency = 650ms") stored efficiently for dashboards and alerting. Logs are per-event records (e.g., "Call 1234: ASR returned 'Tokyo' in 145ms") stored for debugging and audit. Metrics tell you WHAT is happening, logs tell you WHY
Key Metrics to Track
| Metric | Alert Threshold | Critical Threshold |
|---|
| P95 latency | >700ms | >1000ms |
| Error rate | >5% | >10% |
| Call drop rate | >2% | >5% |
| Provider health | 1 provider down | All providers down |
| Active calls | >80% capacity | >95% capacity |
| CPU usage | >70% | >90% |
| Memory usage | >75% | >90% |
| Task completion rate | <80% | <70% |
| Containment rate | <75% | <65% |
| CSAT | <3.5 | <3.0 |
Alerting Rules
# alerting_rules.yaml
groups:
- name: voice_agent_alerts
rules:
- alert: HighLatencyP95
expr: histogram_quantile(0.95, voice_agent_latency_seconds_bucket) > 0.7
for: 5m
labels:
severity: warning
annotations:
summary: "P95 latency above 700ms"
- alert: HighErrorRate
expr: rate(voice_agent_errors_total[5m]) / rate(voice_agent_calls_total[5m]) > 0.05
for: 2m
labels:
severity: critical
annotations:
summary: "Error rate above 5%"
- alert: ProviderDown
expr: voice_agent_provider_health == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Provider {{ $labels.provider }} is down"
- alert: LowTaskCompletion
expr: voice_agent_task_completion_rate < 0.80
for: 10m
labels:
severity: warning
annotations:
summary: "Task completion rate below 80%"
Structured Logging
import structlog
logger = structlog.get_logger()
async def handle_call(call_id, audio_stream):
"""Handle a voice call with structured logging."""
logger.info("call_started",
call_id=call_id,
timestamp=time.time(),
caller_number=mask_number(caller_number),
)
try:
transcript = await asr.transcribe(audio_stream)
logger.info("asr_completed",
call_id=call_id,
provider="deepgram",
latency_ms=145,
word_count=len(transcript.split()),
)
response = await llm.generate(transcript)
logger.info("llm_completed",
call_id=call_id,
provider="openai",
latency_ms=230,
tokens=response.usage.total_tokens,
)
except Exception as e:
logger.error("call_failed",
call_id=call_id,
error=str(e),
error_type=type(e).__name__,
)
raise
Security Considerations
| Security Area | Threat | Mitigation |
|---|
| PII in transcripts | Data leak | Mask PII, encrypt at rest, retention limits |
| API key exposure | Key theft | Environment variables, secrets manager |
| DDoS attacks | Service overload | Rate limiting, WAF, Cloudflare |
| Prompt injection | Malicious input | Input validation, guardrails, system prompt isolation |
| Audio injection | Pre-recorded attacks | Liveness detection, voice biometrics |
| Eavesdropping | Call interception | SRTP (WebRTC), TLS (WebSocket) |
| Data retention | Compliance violation | Auto-delete after 30/90 days, GDPR right to deletion |
| Access control | Unauthorized access | RBAC, audit logs, MFA for admin |
PII Masking
class PIIMasker:
"""Masks PII in transcripts and logs."""
PATTERNS = {
"phone": r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b',
"email": r'\b[\w.-]+@[\w.-]+\.\w+\b',
"credit_card": r'\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b',
"ssn": r'\b\d{3}-\d{2}-\d{4}\b',
"zip": r'\b\d{5}(?:-\d{4})?\b',
}
def mask(self, text):
"""Mask PII in text."""
import re
for pii_type, pattern in self.PATTERNS.items():
text = re.sub(pattern, f'[{pii_type}_MASKED]', text)
return text
Compliance
| Regulation | Requirement | Impact on Voice Agents |
|---|
| GDPR | Right to deletion, data minimization | Delete transcripts on request, minimize data collected |
| CCPA | Right to know, right to delete | Inform callers about recording, honor deletion requests |
| HIPAA | Protected health information | Encrypt transcripts, BAA with providers, audit logs |
| PCI-DSS | Payment card data | Don't store card numbers, tokenize payments |
| TCPA | Consent for recordings | "This call may be recorded" announcement |
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.