Phase 4: Agent Orchestration in Client Environments · 50 min · Python · LangSmith · AgentEvals

Agent Evaluation in Production

The final output is not enough. Evaluate the trajectory.

What you will learn

Perform trajectory evaluation: analyzing the full path of agent actions, not just final output
Measure success rate metrics: task completion rate, tool call accuracy, human approval rate
Track costs: per-task LLM cost, tool call cost, infrastructure cost, total cost per successful task
Use AgentEvals and OpenEvals for agent-specific evaluation
Detect production degradation: agent loops, tool call anomalies, success rate decline

Objective

What you'll learn

This lesson takes approximately 50 min. By the end, you will be able to:

Perform trajectory evaluation: analyzing the full path of agent actions, not just final output
Measure success rate metrics: task completion rate, tool call accuracy, human approval rate
Track costs: per-task LLM cost, tool call cost, infrastructure cost, total cost per successful task
Use AgentEvals and OpenEvals for agent-specific evaluation
Detect production degradation: agent loops, tool call anomalies, success rate decline

The Problem

Evaluating an agent's final output is necessary but not sufficient. You also need to evaluate the trajectory — the sequence of tool calls, reasoning steps, and decisions the agent made to get there. An agent that produces the right answer through a fragile 15-step chain is worse than one that produces it through a robust 3-step chain.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.

Key terms in this lesson

Agent