Phase 7: AI Evaluation & Observability · 55 min · Python · LangSmith · Braintrust

Observability Platforms: LangSmith, Braintrust, HoneyHive, Phoenix

Trace everything. Evaluate continuously. Alert on regressions.

What you will learn

Configure LangSmith: trace capture, LLM-as-judge evaluation, pattern detection, CI/CD integration
Configure Braintrust: eval-first workflow, prompt versioning, Brainstore OLAP, production trace to test case
Configure HoneyHive: agent-specific evaluation, session-level evaluations, agent session annotations
Configure Phoenix: open-source, self-hosted, agent tracing, online evaluation, local deployment
Select platforms based on criteria: self-hosting, agent evaluation depth, CI/CD integration, cost at scale

Objective

What you'll learn

This lesson takes approximately 55 min. By the end, you will be able to:

Configure LangSmith: trace capture, LLM-as-judge evaluation, pattern detection, CI/CD integration
Configure Braintrust: eval-first workflow, prompt versioning, Brainstore OLAP, production trace to test case
Configure HoneyHive: agent-specific evaluation, session-level evaluations, agent session annotations
Configure Phoenix: open-source, self-hosted, agent tracing, online evaluation, local deployment
Select platforms based on criteria: self-hosting, agent evaluation depth, CI/CD integration, cost at scale

The Problem

Trace everything. Evaluate continuously. Alert on regressions. LangSmith, Braintrust, HoneyHive, and Phoenix are the platforms FDEs use to get visibility into LLM applications: token-level traces, retrieval quality metrics, latency breakdowns, and cost tracking. This lesson covers platform selection, integration, and the dashboards that matter.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.