Phase 5: Modern LLM Engineering · 55 min · Python · OpenAI API · Anthropic API

Context Engineering: Designing the Model's Information Environment

Prompt engineering is about what you say. Context engineering is about everything the model sees — and how you structure it for maximum signal.

Hiring signal: Can design production LLM systems that reliably follow instructions by engineering the full context, not just the prompt

What you will learn

Design the full context stack: system prompt, retrieved context, conversation history, user input
Apply context ordering principles to mitigate 'lost in the middle' degradation
Implement token budgeting across context components with priority-based truncation
Structure context with clear delimiters, section headers, and formatting for maximum model comprehension
Build a context assembly pipeline that composes context dynamically per request

The Problem

A team builds a customer support agent. They stuff the system prompt with 3,000 tokens of instructions, dump 5,000 tokens of retrieved documentation, add 2,000 tokens of conversation history, and wonder why the model hallucinates, ignores instructions, and gives inconsistent answers. The context window isn't the problem — the structure of the context is.

Context engineering is the discipline of designing what goes into the context window, in what order, with what priority, and how it's formatted. It's the difference between a model that follows instructions precisely and one that "forgets" your system prompt by the time it reaches the user's question.

The Concept

Beyond Prompt Engineering: The Full Context Stack

Prompt engineering focuses on the text you write — the instructions, the examples, the questions. Context engineering considers the entire information environment the model sees in a single request:

System prompt          → role, rules, output format (always present, always paid for)
Few-shot examples      → demonstrations of desired behavior
Retrieved context      → RAG chunks, database results, API responses
Conversation history   → prior turns (grows over a multi-turn conversation)
Tool results           → output from function calls the model made
User query             → the actual request

Each of these competes for the model's attention budget — a finite resource. The model doesn't read all tokens equally. Research shows LLMs attend more to the beginning and end of context ("lost in the middle" effect), and that instruction-following degrades as context length grows. The art is packing maximum signal into minimum tokens, structured so the model can find and follow what matters.

The Principles of Context Engineering

1. Put instructions first, context last. The system prompt (rules, format, role) should be at the very beginning — this is where attention is strongest. Retrieved context (RAG chunks, tool results) should come near the end, just before the user's query. This leverages the primacy and recency attention bias.

2. Less is more. Every token you add dilutes the attention available to every other token. A 2,000-token system prompt doesn't make the model "more careful" — it makes the model spend attention on instructions instead of the actual task. Be ruthless: cut anything that doesn't directly improve the output.

3. Structure helps attention. XML tags, markdown headers, and numbered sections help the model parse context efficiently. Instead of a wall of text, use <instructions>, <context>, <examples> tags. The model has been trained on structured text and can navigate it more effectively.

4. Separate instructions from data. Don't mix rules ("always respond in JSON") with data (retrieved chunks). The model may confuse a rule in a retrieved document with your actual instructions. Use clear delimiters: <system_rules> vs <retrieved_documents>.

5. Manage conversation history actively. In multi-turn conversations, history grows linearly — every turn adds tokens. Without management, a 20-turn conversation can consume 80% of the context window with history, leaving almost no room for new context. Strategies: summarize older turns, sliding window (keep last N turns), or extract and carry forward only the key facts.

Your customer support agent has a 2,500-token system prompt, retrieves 4,000 tokens of documentation, and has 3,000 tokens of conversation history. The user's query is only 50 tokens. The model is ignoring your instructions. What's the most likely cause and fix?

With 7,000+ tokens of context competing for attention, the model's instruction-following degrades. The system prompt — your rules — is likely buried or mixed with retrieved data. Fix: cut the system prompt to only essential rules, place it at the very start, use XML tags to delimit sections, and summarize or truncate older conversation history to free attention budget.

Context Window vs. Effective Context

A model with a 128K token context window can technically process 128K tokens, but its effective context — the range where it reliably finds and uses information — is much smaller. Research shows:

0-10K tokens: near-perfect retrieval and instruction following
10-50K tokens: good but degrading, especially for details in the middle
50K+ tokens: significant degradation, "lost in the middle" effect dominates

This means that just because you can stuff 100K tokens into the prompt doesn't mean you should. A well-engineered 5K-token context will outperform a bloated 50K-token context on almost every task.

Dynamic Context Assembly

In production, context isn't static — it's assembled dynamically per request based on the query, conversation state, and retrieved information. The assembly pipeline:

1. Start with the system prompt (fixed, always present)
2. Add relevant few-shot examples (selected based on query type)
3. Retrieve and add relevant context (RAG, database, API calls)
4. Add managed conversation history (summarized or windowed)
5. Append the current user query
6. Reserve output space (don't fill the entire window — leave room for the response)

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Ship It, Use It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.