Phase 9: Agent Evaluation & Observability · 60 min · Python · Anthropic SDK · dataclasses
The Problem
Your agent produces the correct final answer. You ship it. A week later, you discover it's costing 4× more than expected per task, and occasionally timing out. You dig in and find it's averaging 24 steps on tasks that should take 6. It loops, calls the wrong tool, calls the right tool multiple times redundantly, then eventually reaches the right answer anyway. Final-answer accuracy told you nothing was wrong.
This is the blind spot of outcome-only evaluation. When you measure only whether the final answer is correct, you miss everything that happened on the way there: how many steps it took, which tools it called, how often it encountered errors, whether it recovered from those errors gracefully, and whether it stated false facts at any point (even if it later self-corrected).
Trajectory evaluation closes this gap. Instead of asking "did the agent get it right?", you ask: "did the agent take a good path to get there?" A trajectory is the full sequence of an agent's actions — every (action, observation) pair from the first tool call to the final answer. Evaluating the trajectory gives you a complete quality signal.
Senior AI engineers at production companies treat trajectory metrics as first-class KPIs alongside final-answer accuracy. Cost efficiency, error recovery rate, and step efficiency all flow directly from trajectory analysis. Without them, you're flying blind on the operational and quality dimensions that determine whether your agent is viable at scale.
What Is a Trajectory?
A trajectory is the complete sequence of an agent's steps: [(input, action_1, observation_1), (action_2, observation_2), ..., (action_N, final_answer)]. Each step is a (action, observation) pair — what the agent decided to do and what happened as a result. For tool-using agents, action = tool_call and observation = tool_result. Trajectory evaluation judges the quality of this entire sequence.
The six trajectory metrics
Each metric captures a distinct dimension of agent quality. Together they give you a complete profile of a single run.
| Metric | Formula | What it catches |
|---|
| Efficiency | optimal_steps / steps_taken | Loops, redundant calls, poor planning |
| Tool accuracy | correct_tool_calls / total_tool_calls | Wrong tool selection, hallucinated tool names |
| Error rate | error_steps / total_steps | API failures, malformed arguments, timeouts |
| Recovery rate | errors_recovered / total_errors | Whether errors cascade or get handled |
| Hallucination rate | hallucinated_steps / total_steps | False factual claims mid-trajectory |
| Cost efficiency | optimal_tokens / actual_tokens | Token waste from loops and redundancy |
Efficiency is the most revealing: an agent with 1.0 accuracy and 0.25 efficiency took 4× the steps it needed to. That's 4× the cost, 4× the latency, and a fragile agent whose self-corrections may not always succeed.
Recovery rate separates resilient agents from fragile ones. In production, some errors are inevitable — API rate limits, timeouts, empty search results. An agent with 90% recovery rate handles these gracefully. One with 20% recovery rate turns every transient error into a task failure.
An agent produces the correct final answer in 24 steps. The optimal trajectory is 6 steps. What does this tell you?
Accuracy alone is a misleading metric. An agent that takes 24 steps when 6 suffice costs 4× as much to run, is 4× slower, and is fragile — its loops or wrong tool calls may not always self-correct. Trajectory efficiency reveals these issues. Optimize for both: correct final answer AND efficient path to get there.
Recording trajectories
The TrajectoryRecorder wraps your agent loop and captures every step. It records tool name, arguments, the observation (tool result or model output), status, token count, and latency for each step.
recorder = TrajectoryRecorder(run_id="run-001", task_description="Research X")
with recorder:
recorder.record_tool_call(
tool_name="web_search",
args={"query": "transformer efficiency 2024"},
observation="Found 8 results...",
status=StepStatus.SUCCESS,
tokens=250,
latency_ms=320,
)
recorder.record_tool_call(
tool_name="read_page",
args={"url": "https://arxiv.org/..."},
observation="Error: 429 Too Many Requests",
status=StepStatus.ERROR,
error_message="429 Too Many Requests",
)
recorder.record_tool_call(
tool_name="read_page",
args={"url": "https://arxiv.org/...", "retry": True},
observation="Page content retrieved successfully.",
status=StepStatus.RECOVERED,
)
recorder.record_final_answer("Transformers have improved efficiency by 3× since 2022.")
trajectory = recorder.trajectory
Three status values: SUCCESS, ERROR, and RECOVERED. The distinction between ERROR and RECOVERED is what powers the recovery rate metric — without it, you can't tell whether errors cascade or get handled.
Building an evaluation dataset
Ground-truth trajectory datasets contain what the optimal path looks like for each task type. Each example specifies: the task description, the optimal step count, the expected tools (the set of tools a correct trajectory should call), and the maximum allowed steps before the run is considered a failure.
dataset = EvaluationDataset([
GroundTruthExample(
task_description="Search for recent papers on transformer efficiency",
optimal_steps=4,
expected_tools=["web_search", "read_page"],
max_allowed_steps=12,
acceptable_efficiency_floor=0.5,
),
# ... 99 more examples
])
Creating high-quality ground-truth data requires expert annotation. For each task type, an experienced engineer traces through the task manually, recording the optimal tool sequence and step count. This is a one-time investment that pays off every time you run evaluation.
Dataset composition guidelines: include simple tasks (2–4 steps), complex multi-step tasks (8–15 steps), error-recovery scenarios (tasks where the agent will encounter a deliberate error), and edge cases (empty search results, tool timeouts, ambiguous queries). A dataset of 100–500 examples is sufficient for statistically meaningful evaluation.
Recovery rate (errors_recovered_from / total_errors) measures what?
Errors in multi-step agents are almost inevitable. What matters is whether the agent can recover from them. Recovery rate distinguishes resilient agents (high recovery rate: "I got an API error, let me try a different approach") from fragile agents (low recovery rate: "I got an error, I'll keep doing the same wrong thing"). Target greater than 80% error recovery rate in production agents.