The Concept
From Notebook to Production: What Changes
A RAG prototype in a Jupyter notebook proves the concept works — you can embed documents, retrieve relevant chunks, and generate answers. But a notebook is not a production system. The gap between "works on my laptop" and "serves 1,000 users reliably" is enormous, and most of the engineering work happens in that gap.
A production RAG system has three independent subsystems, each with its own scaling, failure modes, and operational requirements:
Ingestion pipeline Retrieval API LLM service
↓ ↓ ↓
Parse → chunk → embed Index → query → rerank Prompt → generate
↓ ↓ ↓
Vector store + metadata Filters + auth Observability + cost
Ingestion pipeline runs as a batch job (or streaming job) that processes documents: parsing PDFs/HTML/Word docs, chunking them into pieces, generating embeddings, and writing to a vector store with metadata (source, permissions, timestamps, tags). This pipeline must handle document updates (re-embedding when content changes), deletions (removing stale embeddings), and new documents (incremental indexing). The most common production failure is "new documents aren't showing up in answers" — which almost always means the ingestion pipeline isn't running or isn't re-indexing.
Retrieval API is the service that handles user queries: it takes a query, retrieves candidates from the vector store, applies filters (e.g., "only documents the user has permission to see"), reranks, and returns the top-k chunks. This service must be fast (typically <200ms for retrieval), authenticated, and horizontally scalable. It's stateless, so you can run multiple instances behind a load balancer.
LLM service takes the retrieved context and generates an answer. This is where cost and latency live — LLM calls are the slowest and most expensive part of the pipeline. Production systems add caching (if the same query + retrieved context produces the same answer, return the cached response), rate limiting, fallback models (if the primary model is down, fall back to a cheaper/faster one), and observability (log every call for debugging and cost tracking).
Caching Strategy
Caching is the single most effective cost reduction in production RAG. Many user queries are repeated (especially in customer support — "how do I reset my password?" gets asked thousands of times). A semantic cache (checking if the incoming query is similar to a cached query, not just exact match) can eliminate 30-60% of LLM calls in support use cases. The cache key should be the query + retrieved chunk IDs (not just the query), so that if the underlying documents change, the cache is invalidated automatically.
Your RAG system is in production. Users report that answers about new documents are missing. Which subsystem is most likely failing?
If new documents are missing, the ingestion pipeline is likely not re-indexing updates. Production RAG needs a repeatable, scheduled ingestion and re-indexing process.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Common Pitfalls, Production Checklist, Interview Framing, Evaluation, Exercises, Key Terms — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.