Phase 9: Agent Evaluation & Observability · 60 min · Python · Anthropic SDK · dataclasses

Trajectory Evaluation

The right destination doesn't redeem a terrible path — measure both.

Hiring signal: Trajectory evaluation is what separates engineers who test with examples from those who run systematic quality measurement systems. Mature AI teams at Stripe, Anthropic, and OpenAI partner companies require trajectory-level evaluation before any agent ships.

What you will learn

Define and measure six trajectory evaluation metrics: efficiency, tool accuracy, error rate, recovery rate, hallucination rate, and cost efficiency
Build a TrajectoryRecorder that captures every action/observation step of an agent run
Design a ground-truth evaluation dataset with optimal step counts and expected tool sequences

The Problem

Your agent produces the correct final answer. You ship it. A week later, you discover it's costing 4× more than expected per task, and occasionally timing out. You dig in and find it's averaging 24 steps on tasks that should take 6. It loops, calls the wrong tool, calls the right tool multiple times redundantly, then eventually reaches the right answer anyway. Final-answer accuracy told you nothing was wrong.

This is the blind spot of outcome-only evaluation. When you measure only whether the final answer is correct, you miss everything that happened on the way there: how many steps it took, which tools it called, how often it encountered errors, whether it recovered from those errors gracefully, and whether it stated false facts at any point (even if it later self-corrected).

Trajectory evaluation closes this gap. Instead of asking "did the agent get it right?", you ask: "did the agent take a good path to get there?" A trajectory is the full sequence of an agent's actions — every (action, observation) pair from the first tool call to the final answer. Evaluating the trajectory gives you a complete quality signal.

Senior AI engineers at production companies treat trajectory metrics as first-class KPIs alongside final-answer accuracy. Cost efficiency, error recovery rate, and step efficiency all flow directly from trajectory analysis. Without them, you're flying blind on the operational and quality dimensions that determine whether your agent is viable at scale.

What Is a Trajectory?
A trajectory is the complete sequence of an agent's steps: [(input, action_1, observation_1), (action_2, observation_2), ..., (action_N, final_answer)]. Each step is a (action, observation) pair — what the agent decided to do and what happened as a result. For tool-using agents, action = tool_call and observation = tool_result. Trajectory evaluation judges the quality of this entire sequence.

The Six Trajectory Metrics

Each metric captures a distinct dimension of agent quality. Together they give you a complete profile of a single run.

Metric	Formula	What it catches
Efficiency	optimal_steps / steps_taken	Loops, redundant calls, poor planning
Tool accuracy	correct_tool_calls / total_tool_calls	Wrong tool selection, hallucinated tool names
Error rate	error_steps / total_steps	API failures, malformed arguments, timeouts
Recovery rate	errors_recovered / total_errors	Whether errors cascade or get handled
Hallucination rate	hallucinated_steps / total_steps	False factual claims mid-trajectory
Cost efficiency	optimal_tokens / actual_tokens	Token waste from loops and redundancy

Efficiency is the most revealing: an agent with 1.0 accuracy and 0.25 efficiency took 4× the steps it needed to. That's 4× the cost, 4× the latency, and a fragile agent whose self-corrections may not always succeed.

Recovery rate separates resilient agents from fragile ones. In production, some errors are inevitable — API rate limits, timeouts, empty search results. An agent with 90% recovery rate handles these gracefully. One with 20% recovery rate turns every transient error into a task failure.

An agent produces the correct final answer in 24 steps. The optimal trajectory is 6 steps. What does this tell you?

Accuracy alone is a misleading metric. An agent that takes 24 steps when 6 suffice costs 4× as much to run, is 4× slower, and is fragile — its loops or wrong tool calls may not always self-correct. Trajectory efficiency reveals these issues. Optimize for both: correct final answer AND efficient path to get there.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Recording Trajectories, Building an Evaluation Dataset, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.