Phase 5: Modern LLM Engineering · 55 min · Python · OpenAI API · Vector Database (ChromaDB/Qdrant)

Memory Systems for LLM Agents

A conversation without memory is like talking to someone who forgets everything when they blink. Memory is what makes an agent useful across sessions.

Hiring signal: Can build production LLM agents that maintain state across sessions — a key differentiator between toy demos and real products

What you will learn

Distinguish working memory, episodic memory, and semantic memory for LLM agents
Implement conversation summarization for long-running chat sessions
Build a semantic memory store using embeddings for user preferences and facts
Design a memory retrieval system that surfaces relevant memories at the right time
Handle memory conflicts, staleness, and privacy considerations

The Problem

You build a personal AI assistant. It's brilliant within a single conversation — it remembers what you said 5 minutes ago. But when you come back tomorrow, it asks your name again. It doesn't remember that you prefer Python, that your project uses FastAPI, or that you asked it to review code yesterday. Every session starts from zero.

This is the memory problem: LLMs are stateless. Each API call is independent — the model has no memory of past calls unless you explicitly provide the history in the context window. For simple chatbots, stuffing conversation history into the prompt works. But for agents that need to remember user preferences, project context, past decisions, and evolving facts over days or weeks, you need a memory system.

The Concept

The Three Types of Memory

Human memory has distinct systems for different timescales and purposes. LLM agents need analogous systems:

Working memory (short-term): the current conversation context — what's in the context window right now. This is where the model holds the active thread of conversation, recent tool results, and the current task. It's limited by the context window size and resets when the conversation ends. This is what you've been doing implicitly by passing conversation history in every API call.

Episodic memory (events): records of specific past interactions — "the user asked me to debug a Python script on July 15th" or "I recommended using Pydantic for validation." Episodic memory stores what happened, when, and what the outcome was. In practice, this is a log of past conversations or agent actions, retrieved when relevant to the current task.

Semantic memory (facts): distilled knowledge about the user and the world — "the user prefers Python over JavaScript," "the project uses FastAPI and PostgreSQL," "the user's timezone is PST." Semantic memory is extracted from episodes and stored as durable facts that persist across sessions. This is what makes an agent feel like it "knows" you.

Working memory  : current context window (resets per session)
Episodic memory : log of past interactions (retrieved when relevant)
Semantic memory : distilled facts about user/world (persistent, always available)

How Memory Systems Work in Practice

A production memory system has four operations:

Write: When the user says something worth remembering ("I'm building a React app with TypeScript"), the system extracts the fact and stores it. This can be done with a separate LLM call: "Extract any durable facts from this conversation that would be useful in future sessions."

Read: When a new conversation starts, the system retrieves relevant memories and injects them into the context. The retrieval can be rule-based (always load user preferences), similarity-based (RAG over memory store), or time-based (load facts from the last 7 days).

Update: When facts change ("actually, we switched to Vue"), the system updates the existing memory rather than creating a duplicate. This requires entity resolution — recognizing that "the frontend framework" in the new message refers to the same fact as "React" in the stored memory.

Forget: Memories have a relevance decay. A fact about "the current bug being investigated" is relevant for days, not months. The system should archive or delete stale memories to prevent the memory store from growing unboundedly and diluting retrieval quality.

A user tells your AI assistant "I'm working on a Django project with PostgreSQL." Three weeks later, they ask "how should I set up my database connection?" What type of memory is needed to make the assistant say "since you're using Django with PostgreSQL, use django-db-connections..."?

The fact "user uses Django with PostgreSQL" is a durable fact about the user's preferences/stack — semantic memory. It should be extracted from the conversation, stored persistently, and retrieved when the user asks a database question three weeks later. Without semantic memory, the assistant would give a generic answer instead of a Django-specific one.

Memory Architecture: The Memory Store

The memory store is typically a database (vector database for semantic search, or a simple key-value store for structured facts) that persists across sessions. The architecture:

User message → [Memory Extractor LLM] → extracted facts
                                         ↓
                                    [Memory Store]
                                    (vector DB / KV store)
                                         ↓
New conversation → [Memory Retriever] → relevant memories
                                         ↓
                                    [Context Assembly]
                                    (inject memories into system prompt)
                                         ↓
                                    [LLM generates response]

The key design decisions:

Storage: vector database (for similarity retrieval) vs. structured database (for exact fact lookup) vs. both
Extraction: when to extract (every turn? end of session? only when explicitly asked?) and what to extract (facts only? preferences? decisions?)
Retrieval: how to decide which memories are relevant to the current query (similarity search? keyword match? always load user profile?)
Injection: how to format memories in the context (system prompt section? separate message? XML tags?)

Memory vs. RAG: What's the Difference?

RAG and memory are both retrieval systems that inject information into the context window, but they serve different purposes:

RAG retrieves external knowledge — documentation, articles, databases — that the model doesn't "know." It's about giving the model information it never had.
Memory retrieves the user's own history — past conversations, preferences, decisions — that the model "experienced" but forgot. It's about restoring information the model once had but lost due to statelessness.

In practice, they share infrastructure (both use vector databases and embedding-based retrieval) but differ in lifecycle: RAG documents are relatively static (updated when the knowledge base changes), while memories are continuously written and updated as the user interacts with the system.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Ship It, Use It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.