Phase 9: Agent Evaluation & Observability · 60 min · Python · OpenTelemetry · Langfuse

Distributed Tracing for Agents

You can't debug what you can't see — instrument every span, trace every run.

Hiring signal: Distributed tracing is standard in backend engineering — AI engineers are expected to bring the same rigor to agent systems. Candidates who mention OpenTelemetry, LangSmith, and Langfuse when asked 'how do you debug a production agent failure?' demonstrate real operational maturity.

What you will learn

Instrument a multi-agent system with OpenTelemetry spans using standard LLM semantic conventions
Use Langfuse SDK to trace agent runs with parent-child span relationships capturing cost, latency, and tool metadata
Apply a structured debugging workflow: filter traces by error, inspect span trees, trace causal chains to root cause

The Problem

An agent run fails in production. The error occurs in step 12 of 18. Step 12 called Tool C with an argument that was generated in step 9. Step 9 got its context from a tool result returned in step 4. You can't reproduce the failure because you don't know what step 4 returned. Your logs show the final error message but nothing about the causal chain that led to it.

Without distributed tracing, debugging a multi-step agent is archaeology. You're working backward from an error message, guessing what state the agent was in at each step. With tracing, every step is recorded: what the agent decided to do, what arguments it used, what came back, how long it took, and how much it cost. The trace gives you a complete, queryable record of everything that happened.

This is not a new problem — distributed systems engineers have been solving it with OpenTelemetry for a decade. AI engineers building agents need to apply the same rigor. An agent is a distributed system where the "services" are LLM calls and tool invocations, the "network calls" are model API requests, and the "state" is the accumulated context window. The same tracing tools work for both.

Langfuse and LangSmith are purpose-built trace stores for agent systems. They receive your spans, build visual trace trees, and let you query across thousands of runs to find patterns. They also support online evaluation — running an LLM judge on every production trace automatically, giving you quality scores alongside operational metrics.

Spans and Traces
A span represents one unit of work: one LLM call, one tool call. A trace is a tree of spans linked by a shared trace_id. Every span has a parent_span_id (except the root), a start time, end time, status (OK or ERROR), and a bag of attributes. For agent systems: agent_span → [llm_span, tool_span → [sub_tool_span]]. The tree structure reflects the causal chain.

OpenTelemetry for Agents

OpenTelemetry is the open standard for distributed tracing. The Python SDK provides a Tracer that creates spans. For agent systems, follow the GenAI semantic conventions — standard attribute names that tracing backends understand.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

# Setup
provider = TracerProvider(resource=Resource.create({"service.name": "my-agent"}))
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("my-agent")

# Standard LLM attributes (OTel GenAI semantic conventions)
# gen_ai.operation.name  = "chat" | "embeddings" | "completion"
# gen_ai.request.model   = "claude-sonnet-4-5"
# gen_ai.usage.input_tokens  = 1234
# gen_ai.usage.output_tokens = 567
# gen_ai.system          = "anthropic" | "openai"

# Custom agent attributes
# agent.tool.name   = "web_search"
# agent.tool.args   = '{"query": "..."}'
# agent.cost_usd    = 0.000456

A span for a tool call:

with tracer.start_as_current_span("web_search") as span:
    span.set_attribute("agent.tool.name", "web_search")
    span.set_attribute("agent.tool.args", json.dumps({"query": query}))
    span.set_attribute("gen_ai.request.model", "none")  # tool, not LLM
    result = web_search_tool(query)
    span.set_attribute("agent.tool.result", result[:500])  # truncate long results
    span.set_attribute("agent.latency_ms", elapsed_ms)
    if result.is_error:
        span.set_status(StatusCode.ERROR, result.error_message)

The parent-child hierarchy is automatic: if you start a span while another span is active (via context manager nesting), the inner span becomes a child. The trace tree emerges naturally from your code structure.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Langfuse Integration, The Debugging Workflow, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.