The Concept
What Is a Context Window and Why Does It Cost Money?
Every LLM has a context window — the maximum number of tokens it can process in a single request. This includes the system prompt, any few-shot examples, retrieved context (from RAG), the user's query, and the model's output. A model with a 128K token context window sounds like it can handle anything, but every token you put in costs money and affects quality.
The cost model is straightforward but often ignored: you pay for input tokens (what you send to the model) and output tokens (what the model generates). Input tokens are typically 2-5x cheaper than output tokens, but in RAG systems, input tokens dominate because you're stuffing large chunks of retrieved text into every request. If you retrieve 20 chunks of 400 tokens each, that's 8,000 input tokens per query — before the model has generated a single word of output.
The quality impact is less obvious but equally important. Research from Stanford (the "Lost in the Middle" paper, Liu et al. 2023) showed that LLMs pay less attention to information in the middle of long contexts. They attend strongly to the beginning and end of the input, but information buried in the middle gets ignored. This means that stuffing more context can actually degrade performance — the model has to find the relevant information among noise, and it's less likely to find it if it's in the middle of a 50,000-token blob.
The Context Budget
Think of the context window as a budget you allocate across different components:
Total budget: input_tokens + output_tokens ≤ context_window Actual cost: (input_tokens × input_price) + (output_tokens × output_price)
Every token in the system prompt is paid on every request. Every few-shot example is paid on every request. Every retrieved chunk is paid on every request. The question is not "can it fit?" but "is each token earning its place by improving the output?"
The "Lost in the Middle" Problem
Research shows models pay less attention to information in the middle of long contexts:
The practical implication: put the most important information at the START and END of context. If you're building a RAG system, place the most relevant chunk first (start position) and the second most relevant last (end position). Don't just dump chunks in retrieval-score order — reorder for attention.
Token Economics
| Model | Input $/1M tokens | Output $/1M tokens | 1000 req/day (500 in, 200 out) |
|---|
| GPT-4o | $2.50 | $10.00 | $3.25/day |
| GPT-4o-mini | $0.15 | $0.60 | $0.20/day |
| Claude Sonnet | $3.00 | $15.00 | $4.50/day |
| Claude Haiku | $0.80 | $4.00 | $1.20/day |
Key insight: Input tokens are 2-5x cheaper than output tokens, but in RAG systems, input volume is 10-40x larger than output. Compress inputs (fewer chunks, smaller chunks, summarization) for the biggest cost savings. Compress outputs (shorter responses, structured formats) for quality and latency improvements.
Your RAG system retrieves 20 chunks (8,000 tokens) for every query, but most queries only need 3-4 chunks. Your monthly API bill is $15,000. What's the likely impact of reducing to top-5 chunks with reranking?
Most RAG queries are answerable from 3-5 high-quality chunks. The extra 15 chunks add token cost (input billing), increase latency (longer context = slower inference), and dilute signal (the model has to find the relevant info among noise). Reranking to select top-5 typically maintains or improves answer quality while cutting cost ~60%.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Common Pitfalls, Evaluation, Exercises, Key Terms, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.