Phase 9: LLMOps & Production Deployment · 70 min · FastAPI · vLLM · Modal

Model Serving & Inference Optimization

A model that takes 10 seconds to respond is a model nobody uses. Optimize for the user, not the benchmark.

Hiring signal: Production inference engineering — latency, throughput, cost optimization

The Problem

A team deploys their RAG system. It works in testing. In production with 50 concurrent users, latency jumps from 500ms to 8 seconds. The embedding model is the bottleneck — it processes one request at a time. The LLM calls are sequential, not parallel. There is no caching for repeated queries. The system costs $200/day because every request hits the most expensive model.

Inference optimization is not premature optimization. It is the difference between a product that scales and one that collapses under real traffic.

The Concept

Inference Optimization Layers

Your LLM API receives 10,000 queries/day, but 30% are repeats of common questions ("What is your return policy?", "How do I reset my password?"). Each costs $0.02 in API fees. What optimization layer saves the most money?

Caching (Layer 4) is the highest-ROI optimization here. 3,000 daily repeat queries can be served from cache with zero latency and zero API cost. Semantic caching (embedding similarity) catches paraphrased repeats too. No model change, no quality degradation, pure cost savings. Start with caching before quantization or routing.

Key Metrics for Serving

Metric	What It Measures	Target
TTFT (Time To First Token)	How fast the response starts streaming	< 200ms
TPS (Tokens Per Second)	Generation speed	> 30 tok/s
P95 latency	Worst-case user experience	< 2s for simple queries
Throughput	Requests per second at target latency	Depends on scale
Cost per request	$/request or $/1k tokens	Track and optimize
GPU utilization	How efficiently hardware is used	> 70%

Serving Patterns

Pattern	When to Use	Trade-off
API proxy (OpenAI, Anthropic)	Starting out, variable load	Simplest, but no control
Serverless (Modal, Replicate)	Bursty traffic, GPU needed	Cold starts, but auto-scales
Dedicated GPU (vLLM on EC2)	Consistent high volume	Most control, but expensive idle
Edge/local (Ollama, llama.cpp)	Privacy, low latency	Limited model size

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.