Phase 7: Production, Monitoring & Capstone · 50 min · Langfuse · LangSmith · n8n

Production Monitoring with Langfuse and LangSmith

If you can't see it, you can't fix it.

Hiring signal: Production monitoring is the skill that separates prototype builders from production engineers. Being able to set up Langfuse tracing, define alert thresholds, and explain what metrics to monitor (latency, cost, error rate, faithfulness) demonstrates the operational maturity that production AI roles require.

What you will learn

Set up Langfuse for LLM observability: tracing, metrics, and dashboards
Monitor key metrics: latency, cost per execution, error rate, token usage, faithfulness scores
Configure alerts: threshold-based notifications for cost spikes, error rate increases, latency degradation
Compare Langfuse (open-source, self-hosted) with LangSmith (LangChain's managed offering)

The Problem

Your AI workflow is in production. It processes 500 emails per day. Most work fine, but some fail — and you don't know which ones, why they failed, or how much they cost. When your manager asks "What's our error rate?" and "How much are we spending on LLM calls?", you can't answer. You're flying blind.

Production monitoring fixes this. Langfuse (open-source, self-hosted) and LangSmith (LangChain's managed offering) provide tracing, metrics, and dashboards for LLM applications. They show you every execution, every LLM call, every cost, and every error — in real time.

Production monitoring is the difference between "it works" and "I know it works"
Without monitoring, you discover problems when customers complain. With monitoring, you discover problems before customers notice — and you can answer any question about cost, latency, error rate, or quality with data.

The Concept

What to Monitor

┌──────────────────────────────────────────────────────────────┐
│  PRODUCTION AI MONITORING METRICS                             │
│                                                              │
│  PERFORMANCE                                                  │
│  ├── Latency: time from trigger to completion (p50, p95)    │
│  ├── Throughput: executions per minute/hour/day             │
│  └── Token usage: input tokens, output tokens per execution │
│                                                              │
│  COST                                                        │
│  ├── Cost per execution: LLM + embedding + reranking        │
│  ├── Daily/monthly cost: aggregate spending                 │
│  └── Cost by model: which models consume the most budget    │
│                                                              │
│  QUALITY                                                     │
│  ├── Error rate: % of executions that fail                  │
│  ├── Faithfulness: is the answer grounded in context?       │
│  └── Auto-process rate: % completed without human review    │
│                                                              │
│  USAGE                                                       │
│  ├── Execution count: total per day/week                    │
│  ├── Tool calls: which tools are called most/least          │
│  └── Fallback rate: % routed to human review                │
└──────────────────────────────────────────────────────────────┘

Langfuse vs LangSmith

Aspect	Langfuse	LangSmith
Type	Open-source, self-hosted	Managed (LangChain)
Cost	Free (self-hosted)	Free tier, then paid
Setup	Docker Compose	API key (instant)
Tracing	Full LLM tracing	Full LLM tracing
Dashboards	Built-in	Built-in
Alerts	Webhook-based	Email/Slack
Integrations	OpenAI, LangChain, n8n	LangChain ecosystem
Best for	Data privacy, cost control	LangChain users, quick setup

Your company processes customer emails with AI and must comply with GDPR. Customer email content cannot leave your infrastructure. Which monitoring tool should you use?

Langfuse is the right choice because it's self-hosted via Docker Compose. All tracing data (including customer email content) stays on your infrastructure. LangSmith is managed by LangChain — tracing data (including email content in prompts and responses) would be sent to LangChain's servers, which may violate GDPR data residency requirements. Langfuse gives you full control over data retention, encryption, and access.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Exercises, Key Terms, Common Pitfalls — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.