Phase 9: Evaluation, Production & Capstone · 50 min · Python · Prometheus · Grafana

Monitoring & Observability

Generation success rates, quality scores, GPU utilization, costs — if you can't see it, you can't operate it.

Hiring signal: Monitoring and observability for generative media systems (success rates, quality trends, GPU utilization, cost tracking) is tested in production operations interviews.

What you will learn

Track key metrics: generation success rate, average quality score, P95 generation latency, GPU utilization
Monitor quality score trends over time to detect model drift
Monitor API health: provider uptime, rate limit usage, error rates by provider
Implement cost tracking: per-generation cost, daily/monthly spend, budget alerts

The Problem

A generative media platform runs 10,000 generations per day. Without monitoring:

Is the success rate 95% or 60%? (you don't know)
Is quality degrading over time? (you don't know)
Are you over budget? (you find out at month-end)
Is GPU utilization 20% or 80%? (you don't know)

Monitoring and observability transform a black-box system into an operated system.

What you'll build
Track key metrics (success rate, quality scores, P95 latency, GPU utilization), monitor quality trends for model drift, monitor API health (provider uptime, rate limits, errors), and implement cost tracking with budget alerts.

Key Metrics

Generation Metrics

from dataclasses import dataclass, field
from collections import defaultdict
import time

@dataclass
class GenerationEvent:
    """A single generation event."""
    timestamp: float
    provider: str
    model: str
    status: str  # success, failed, timeout, content_policy
    latency_s: float
    quality_score: float
    cost: float

class MetricsCollector:
    """Collect and aggregate generation metrics."""
    
    def __init__(self):
        self.events: list[GenerationEvent] = []
    
    def record(self, event: GenerationEvent):
        """Record a generation event."""
        self.events.append(event)
    
    def get_success_rate(self, window_hours: int = 24) -> float:
        """Calculate success rate over time window."""
        cutoff = time.time() - window_hours * 3600
        recent = [e for e in self.events if e.timestamp > cutoff]
        if not recent:
            return 0.0
        successes = sum(1 for e in recent if e.status == "success")
        return successes / len(recent) * 100
    
    def get_avg_quality(self, window_hours: int = 24) -> float:
        """Average quality score over time window."""
        cutoff = time.time() - window_hours * 3600
        recent = [e for e in self.events if e.timestamp > cutoff and e.status == "success"]
        if not recent:
            return 0.0
        return sum(e.quality_score for e in recent) / len(recent)
    
    def get_p95_latency(self, window_hours: int = 24) -> float:
        """P95 latency over time window."""
        cutoff = time.time() - window_hours * 3600
        recent = [e.latency_s for e in self.events if e.timestamp > cutoff]
        if not recent:
            return 0.0
        recent.sort()
        p95_idx = int(len(recent) * 0.95)
        return recent[p95_idx]
    
    def get_total_cost(self, window_hours: int = 24) -> float:
        """Total cost over time window."""
        cutoff = time.time() - window_hours * 3600
        return sum(e.cost for e in self.events if e.timestamp > cutoff)
    
    def get_error_breakdown(self, window_hours: int = 24) -> dict:
        """Error breakdown by type."""
        cutoff = time.time() - window_hours * 3600
        recent = [e for e in self.events if e.timestamp > cutoff and e.status != "success"]
        breakdown = defaultdict(int)
        for e in recent:
            breakdown[e.status] += 1
        return dict(breakdown)

Metric Categories

Category	Metrics	Alert Threshold
Generation	Success rate, error breakdown	Success < 90%
Quality	Avg quality score, quality trend	Score drops 10%
Latency	P50, P95, P99 generation time	P95 > 30s
Cost	Daily/monthly spend, cost per generation	Daily > $100
GPU	Utilization, VRAM usage, cold starts	Util < 20% (waste)
API Health	Provider uptime, rate limit usage	Uptime < 99%

Why is tracking P95 latency more useful than average latency for production monitoring?

P95 is easier to compute

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Quality Trend Monitoring, API Health Monitoring, Cost Tracking, Prometheus Integration, Grafana Dashboard, Key Takeaways, What's Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Latency