Phase 9: Agent Evaluation & Observability · 60 min · Python · OpenTelemetry · Langfuse
The Problem
An agent run fails in production. The error occurs in step 12 of 18. Step 12 called Tool C with an argument that was generated in step 9. Step 9 got its context from a tool result returned in step 4. You can't reproduce the failure because you don't know what step 4 returned. Your logs show the final error message but nothing about the causal chain that led to it.
Without distributed tracing, debugging a multi-step agent is archaeology. You're working backward from an error message, guessing what state the agent was in at each step. With tracing, every step is recorded: what the agent decided to do, what arguments it used, what came back, how long it took, and how much it cost. The trace gives you a complete, queryable record of everything that happened.
This is not a new problem — distributed systems engineers have been solving it with OpenTelemetry for a decade. AI engineers building agents need to apply the same rigor. An agent is a distributed system where the "services" are LLM calls and tool invocations, the "network calls" are model API requests, and the "state" is the accumulated context window. The same tracing tools work for both.
Langfuse and LangSmith are purpose-built trace stores for agent systems. They receive your spans, build visual trace trees, and let you query across thousands of runs to find patterns. They also support online evaluation — running an LLM judge on every production trace automatically, giving you quality scores alongside operational metrics.
Spans and Traces
A span represents one unit of work: one LLM call, one tool call. A trace is a tree of spans linked by a shared trace_id. Every span has a parent_span_id (except the root), a start time, end time, status (OK or ERROR), and a bag of attributes. For agent systems: agent_span → [llm_span, tool_span → [sub_tool_span]]. The tree structure reflects the causal chain.
OpenTelemetry for agents
OpenTelemetry is the open standard for distributed tracing. The Python SDK provides a Tracer that creates spans. For agent systems, follow the GenAI semantic conventions — standard attribute names that tracing backends understand.
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
# Setup
provider = TracerProvider(resource=Resource.create({"service.name": "my-agent"}))
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("my-agent")
# Standard LLM attributes (OTel GenAI semantic conventions)
# gen_ai.operation.name = "chat" | "embeddings" | "completion"
# gen_ai.request.model = "claude-sonnet-4-5"
# gen_ai.usage.input_tokens = 1234
# gen_ai.usage.output_tokens = 567
# gen_ai.system = "anthropic" | "openai"
# Custom agent attributes
# agent.tool.name = "web_search"
# agent.tool.args = '{"query": "..."}'
# agent.cost_usd = 0.000456
A span for a tool call:
with tracer.start_as_current_span("web_search") as span:
span.set_attribute("agent.tool.name", "web_search")
span.set_attribute("agent.tool.args", json.dumps({"query": query}))
span.set_attribute("gen_ai.request.model", "none") # tool, not LLM
result = web_search_tool(query)
span.set_attribute("agent.tool.result", result[:500]) # truncate long results
span.set_attribute("agent.latency_ms", elapsed_ms)
if result.is_error:
span.set_status(StatusCode.ERROR, result.error_message)
The parent-child hierarchy is automatic: if you start a span while another span is active (via context manager nesting), the inner span becomes a child. The trace tree emerges naturally from your code structure.
Langfuse integration
Langfuse is an open-source tracing platform with a managed cloud offering and self-hosted Docker option. The Python SDK provides a context manager pattern that maps well to agent loops.
from langfuse import Langfuse
client = Langfuse(
secret_key=os.getenv("LANGFUSE_SECRET_KEY"),
public_key=os.getenv("LANGFUSE_PUBLIC_KEY"),
)
# Top-level trace for one agent run
trace = client.trace(name="research_agent", metadata={"query": query})
# Child spans for each step
planning_span = trace.span(name="planning", input={"query": query})
# ... LLM call ...
planning_span.end(output={"plan": plan}, usage={"input": 300, "output": 200})
search_span = trace.span(name="web_search", input={"query": query})
# ... tool call ...
search_span.end(output={"results": results})
client.flush() # ensure all spans are sent before the process exits
LangSmith uses a decorator pattern that instruments entire functions automatically:
from langsmith import traceable
@traceable(name="research_agent", run_type="chain")
def research_agent(query: str) -> str:
plan = plan_research(query) # auto-traced as child span
results = search_web(query) # auto-traced
answer = synthesize(plan, results) # auto-traced
return answer
Both backends provide: a visual span tree viewer, filtering by latency/cost/error, dataset management (save traces as eval examples), and online evaluation (attach an LLM judge to run on every incoming trace).
A trace shows: agent_span → research_span → [search_tool_span, read_tool_span] → synthesis_span. The synthesis_span shows an error. Where should you look first?
In a causal trace, errors at later steps are usually caused by bad inputs from earlier steps. If synthesis_span failed, check what inputs it received: inspect search_tool_span and read_tool_span results. Was the search tool result empty? Did read_tool return an error page? The trace shows the full causal chain — follow it upstream from the failing span. This is the same root-cause analysis pattern used in distributed backend debugging.
The debugging workflow
A structured 6-step process for debugging agent failures using traces:
Step 1 — Filter traces by error status or unexpected output. Langfuse query: status = ERROR. LangSmith: filter by run type and error.
Step 2 — Open the failing trace, expand the span tree. Look for red/errored spans.
Step 3 — Identify the failing span. Note its position in the tree — is it a tool span or an LLM reasoning span?
Step 4 — Inspect the failing span's inputs. What did the agent pass to this span? What context did it receive from parent spans? Langfuse shows the full input and output for every span.
Step 5 — Identify root cause. Was the error in the inputs (bad data from an upstream span) or in the span itself (model failure, tool bug, network issue)?
Step 6 — Fix and re-run using the logged inputs. Langfuse and LangSmith both let you re-run a specific trace with the same inputs — critical for verifying your fix.
This workflow turns "I have no idea why it failed" into a structured investigation that takes minutes, not hours.
"Online evaluation" in LangSmith and Langfuse means:
Online evaluation is the bridge between offline test-set evaluation and production monitoring. Configure: for every trace that arrives, run an LLM-as-judge on the output. The scores flow into dashboards — you can see quality trends over time, catch degradation from model updates, and correlate quality scores with operational metrics. This is how you know your agent's quality in production, not just on your test set.