Phase 8: Evaluation, Testing & Production · 50 min · Python · Hamming.ai · Cekura

Testing Frameworks: Hamming, Cekura, Coval

Automated testing is what separates production voice agents from demos. Test every change.

Hiring signal: Automated test suite design for voice agents shows production engineering maturity.

What you will learn

Use Hamming.ai for automated testing, regression testing, and CI/CD integration
Use Cekura for conversation simulation and test case generation
Use Coval for STT/TTS benchmarking and provider comparison
Build a custom test framework with call recordings and evaluation scripts

The Problem

A voice agent has infinite conversation paths. Manual testing covers maybe 20 scenarios. But what about the edge cases — angry callers, unexpected questions, accents, background noise, tool failures? You need automated testing frameworks that simulate calls at scale and evaluate quality.

The Concept

Testing Pipeline

Voice Agent Testing Layers

Layer	What's Tested	Tools
Unit	Individual functions (slot filling, intent parsing)	pytest
Integration	ASR → LLM → TTS pipeline	Custom scripts
Conversation	Full call simulation with scenarios	Hamming, Cekura, Coval
Load	Concurrent calls, scaling	Locust, k6
Production	Real calls, shadow testing	Monitoring

Testing Frameworks Comparison

Framework	Type	Approach	Best For
Hamming	Conversation simulation	Phone call emulation with eval	End-to-end call testing
Cekura	AI-powered testing	LLM-generated test cases	Coverage at scale
Coval	Evaluation platform	Predefined + dynamic scenarios	Enterprise QA

Why can't you rely only on unit tests for a voice agent?

Unit tests test individual functions but miss integration issues — ASR mishearing words, LLM generating off-topic responses, TTS cutting off mid-sentence, VAD triggering too early. Voice agents are pipelines where errors compound across components

Hamming: Conversation Simulation

Hamming simulates phone calls to your voice agent and evaluates responses:

from hamming import HammingTester, Scenario

tester = HammingTester(api_key="your-key")

# Define test scenario
scenario = Scenario(
    name="Book flight to Tokyo",
    steps=[
        {"user": "I want to book a flight to Tokyo"},
        {"expect": "ask for date"},
        {"user": "July 15th"},
        {"expect": "ask for passengers or confirm"},
        {"user": "2 passengers"},
        {"expect": "confirm booking"},
    ],
    success_criteria={
        "task_completed": True,
        "mentioned_destination": "Tokyo",
        "mentioned_date": "July 15",
    }
)

results = await tester.run_scenario(scenario)

Cekura: AI-Powered Test Generation

Cekura uses LLMs to generate test cases from your agent's description:

from cekura import CekuraClient

client = CekuraClient(api_key="your-key")

# Generate test cases from agent description
test_suite = await client.generate_tests(
    agent_description="Airline booking voice agent",
    capabilities=["book flights", "check status", "cancel booking"],
    edge_cases=["angry caller", "unclear speech", "change mind mid-booking"],
    num_tests=100
)

# Run all tests
results = await client.run_suite(test_suite)

Coval: Enterprise Evaluation

Coval provides predefined and dynamic evaluation scenarios:

from coval import CovalEvaluator

evaluator = CovalEvaluator(api_key="your-key")

# Define evaluation criteria
criteria = {
    "task_completion": True,
    "response_relevance": True,
    "latency_p95_ms": 700,
    "talk_ratio_max": 60,
    "no_hallucination": True,
    "appropriate_tone": True,
}

# Run evaluation
results = await evaluator.evaluate(
    agent_endpoint="wss://your-agent.com/socket",
    scenarios=["booking", "cancellation", "status_check", "escalation"],
    criteria=criteria
)

Test Scenario Design

# test_scenarios.yaml
scenarios:
  - name: "Successful booking"
    user_inputs:
      - "I want to book a flight to Tokyo"
      - "July 15th"
      - "2 passengers"
      - "Yes, that's correct"
    success_criteria:
      task_completed: true
      destination: "Tokyo"
      date: "July 15"
      passengers: 2

  - name: "Caller changes mind"
    user_inputs:
      - "I want to book a flight to Paris"
      - "Actually, change that to Tokyo"
      - "July 15th"
    success_criteria:
      task_completed: true
      destination: "Tokyo"  # Not Paris
      handled_change: true

  - name: "Angry caller"
    user_inputs:
      - "I've been on hold for 30 minutes!"
      - "This is ridiculous"
    success_criteria:
      de_escalated: true
      no_apology_overload: true  # Don't apologize more than once

  - name: "Tool failure recovery"
    user_inputs:
      - "Check flight DV2487 status"
    success_criteria:
      graceful_error: true  # Agent handles tool failure gracefully
      offers_alternative: true

Evaluation Criteria

Criterion	How to Evaluate	Pass Threshold
Task completion	All slots filled, goal achieved	>85%
Response relevance	Response addresses user's question	>90%
No hallucination	Agent doesn't fabricate information	100%
Appropriate tone	Professional, not robotic	Manual + LLM judge
Latency P95	95th percentile response time	<700ms
Talk ratio	Agent doesn't dominate	<60%
Barge-in handling	Agent stops when interrupted	100%
Error recovery	Agent handles tool failures	>80%

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.