The Problem
You use the same AI assistant every day for 6 months. It still asks for your name. It still recommends the same things you told it you don't like. It has no memory.
Meanwhile, your best human collaborator remembers your preferences, your past decisions, and the context of ongoing projects — without you re-explaining. This lesson is the architecture that makes AI remember like a person.
The insight: human memory is not one system. It's four. Each handles a different kind of information with different retrieval characteristics. Agent memory works the same way.
Four memory types
In-context memory (working memory): Everything in the current context window. Fast (zero latency), but lost at session end. Capacity: 100k–200k tokens for modern models. Use for: the current conversation, retrieved documents, temporary computation results.
Episodic memory: Records of specific past events. "On 2025-11-14, user asked about deployment options and we decided on Vercel." Stored in a vector DB or key-value store. Retrieval: semantic search over past episodes. Permanent until explicitly deleted.
Semantic memory: General knowledge, facts, user preferences. "User prefers Python over JavaScript. User works in the healthcare domain. User dislikes verbose explanations." Stored as structured facts. Retrieval: direct lookup or similarity search.
Procedural memory: How to do things. Stored as tool definitions, system prompt rules, or fine-tuned model weights. Slowest to update — requires redeployment or retraining. Most durable — doesn't fade or get stale the way episodic memories do.
Memory type selection guide
The decision is straightforward once you characterize the information:
| Information type | Memory type | Example |
|---|
| Current conversation | In-context | All messages in this session |
| Past specific interactions | Episodic | "Last week we debugged the auth issue" |
| Stable user facts | Semantic | "Prefers brevity", "expert in ML" |
| Behavioral patterns | Procedural | System prompt rule: "Always use bullet points" |
| Retrieved documents | In-context (injected) | Relevant past episodes from vector DB |
Most production agents combine three of the four: in-context (always) + episodic (past events) + semantic (user profile). Procedural is updated infrequently — it changes when you update your system prompt or retrain the model.
A user told your agent "I prefer bullet points over paragraphs" in a conversation 2 months ago. Which memory type should store this?
"Prefers bullet points" is a stable fact about the user — it belongs in semantic memory (user profile). Episodic would store "2 months ago the user said X" — but what the agent needs isn't the episode, it's the derived fact. Semantic memory stores the extracted, actionable fact. In-context: you'd inject this from the semantic store at the start of each conversation.
Tradeoffs
Each memory type has a cost structure:
| Memory type | Latency | Capacity | Durability | Update cost |
|---|
| In-context | Zero | ~200K tokens | None (session only) | Free (just add to context) |
| Episodic | Vector search (~50ms) | Unlimited | High | LLM extraction + embed |
| Semantic | Direct lookup (~5ms) | Schema-bound | High | Manual or LLM extraction |
| Procedural | Zero (in prompt) | Small (100-500 tokens) | Very high | Redeployment |
The key tradeoff is retrieval latency vs. capacity vs. durability:
- In-context wins on latency but loses on capacity and durability
- Semantic wins on lookup speed but requires a structured schema
- Episodic is the most flexible but adds retrieval latency on every turn
- Procedural is the most durable but the hardest to update without engineering work
A user's ongoing project has a 200-message history across 5 sessions. The agent needs to reference key decisions from session 1 when answering a question in session 5. Which memory approach works best?
200 messages at ~200 tokens each = 40,000 tokens — expensive to load every time, and most of it is irrelevant to any given question. Episodic memory stores condensed episode summaries (key decisions, outcomes, open questions). Session 5 retrieves only the 2–3 relevant past episodes. This is how long-running project assistants work in production.
Build
Build it
Before the full implementation, here's a minimal illustration of in-context memory as a TTL (time-to-live) cache — items expire and get evicted once they age past a limit, the same way working memory fades once a session ends.
import time
import random
class TTLMemoryCache:
"""A dict-based cache where entries expire after `ttl` seconds."""
def __init__(self, ttl_seconds):
self.ttl = ttl_seconds
self.store = {} # key -> (value, inserted_at)
def set(self, key, value, now):
self.store[key] = (value, now)
def get(self, key, now):
if key not in self.store:
return None
value, inserted_at = self.store[key]
if now - inserted_at > self.ttl:
del self.store[key]
return None
return value
def active_keys(self, now):
return [k for k, (_, t) in self.store.items() if now - t <= self.ttl]
cache = TTLMemoryCache(ttl_seconds=5)
clock = 0
cache.set("last_topic", "PostgreSQL vs SQLite", clock)
cache.set("user_name", "Alex", clock)
print(f"t={clock}: active memory = {cache.active_keys(clock)}")
clock = 3
cache.set("current_file", "auth.py", clock)
print(f"t={clock}: active memory = {cache.active_keys(clock)}")
clock = 6
print(f"t={clock}: 'last_topic' still there? {cache.get('last_topic', clock)}")
print(f"t={clock}: active memory = {cache.active_keys(clock)}")
The following implementation provides a unified `MemorySystem` interface with all four memory types. Each type has its own storage and retrieval semantics.
"""
lesson_07_01_memory_taxonomy.py
Unified memory system with all four memory types:
- InContextMemory: message history with token budget
- EpisodicMemory: past events with semantic retrieval
- SemanticMemory: structured user facts
- ProceduralMemory: system prompt rules registry
"""
import os
import json
import time
import math
from dataclasses import dataclass, field
from typing import Any
import anthropic
@dataclass
class Message:
role: str # "user" | "assistant" | "system"
content: str
timestamp: float = field(default_factory=time.time)
token_count: int = 0
class InContextMemory:
"""Working memory: the current context window."""
def __init__(self, max_tokens: int = 40000):
self.max_tokens = max_tokens
self.messages: list[Message] = []
self._token_count = 0
def add(self, role: str, content: str) -> None:
tokens = len(content) // 4 # rough estimate
msg = Message(role=role, content=content, token_count=tokens)
self.messages.append(msg)
self._token_count += tokens
# Simple eviction: remove oldest messages when over budget
while self._token_count > self.max_tokens and len(self.messages) > 1:
removed = self.messages.pop(0)
self._token_count -= removed.token_count
def as_list(self) -> list[dict[str, str]]:
return [{"role": m.role, "content": m.content} for m in self.messages]
@property
def token_usage(self) -> dict[str, int]:
return {"used": self._token_count, "budget": self.max_tokens,
"remaining": self.max_tokens - self._token_count}
@dataclass
class Episode:
"""A stored past event."""
episode_id: str
text: str
timestamp: float
user_id: str
session_id: str
tags: list[str]
importance: float # 1.0–5.0
embedding: list[float] | None = None
class EpisodicMemory:
"""Stores and retrieves past events by semantic similarity."""
def __init__(self):
self.episodes: dict[str, Episode] = {}
self._next_id = 0
def store(self, text: str, user_id: str, session_id: str,
tags: list[str] | None = None, importance: float = 3.0) -> str:
episode_id = f"ep_{self._next_id:04d}"
self._next_id += 1
self.episodes[episode_id] = Episode(
episode_id=episode_id, text=text, timestamp=time.time(),
user_id=user_id, session_id=session_id,
tags=tags or [], importance=importance,
)
return episode_id
def retrieve(self, query: str, user_id: str, top_k: int = 5) -> list[Episode]:
"""Simple keyword-based retrieval (use vector DB in production)."""
query_words = set(query.lower().split())
user_episodes = [e for e in self.episodes.values() if e.user_id == user_id]
scored = []
for ep in user_episodes:
ep_words = set(ep.text.lower().split())
overlap = len(query_words & ep_words) / max(len(query_words), 1)
recency = math.exp(-0.01 * (time.time() - ep.timestamp) / 3600)
score = 0.7 * overlap + 0.2 * recency + 0.1 * (ep.importance / 5.0)
scored.append((score, ep))
scored.sort(key=lambda x: x[0], reverse=True)
return [ep for _, ep in scored[:top_k]]
def count(self, user_id: str) -> int:
return sum(1 for e in self.episodes.values() if e.user_id == user_id)
@dataclass
class UserProfile:
"""Structured semantic memory for one user."""
user_id: str
name: str = ""
expertise_level: str = "unknown" # novice | intermediate | expert
preferred_language: str = "Python"
communication_preferences: dict[str, bool] = field(default_factory=dict)
domain_focus: list[str] = field(default_factory=list)
explicit_preferences: dict[str, Any] = field(default_factory=dict)
last_updated: float = field(default_factory=time.time)
def update(self, key: str, value: Any) -> None:
if hasattr(self, key):
setattr(self, key, value)
else:
self.explicit_preferences[key] = value
self.last_updated = time.time()
def to_context_string(self) -> str:
parts = [f"User: {self.name}" if self.name else ""]
if self.expertise_level != "unknown":
parts.append(f"Expertise: {self.expertise_level}")
if self.preferred_language:
parts.append(f"Preferred language: {self.preferred_language}")
if self.domain_focus:
parts.append(f"Domain focus: {', '.join(self.domain_focus)}")
for k, v in self.explicit_preferences.items():
parts.append(f"{k}: {v}")
return " | ".join(p for p in parts if p)
class SemanticMemory:
"""Structured facts about users and domain knowledge."""
def __init__(self):
self.profiles: dict[str, UserProfile] = {}
self.facts: dict[str, dict[str, Any]] = {} # user_id → {key: value}
def get_profile(self, user_id: str) -> UserProfile:
if user_id not in self.profiles:
self.profiles[user_id] = UserProfile(user_id=user_id)
return self.profiles[user_id]
def set_fact(self, user_id: str, key: str, value: Any) -> None:
if user_id not in self.facts:
self.facts[user_id] = {}
self.facts[user_id][key] = value
def get_fact(self, user_id: str, key: str, default: Any = None) -> Any:
return self.facts.get(user_id, {}).get(key, default)
@dataclass
class ProceduralRule:
"""A system prompt rule."""
rule_id: str
description: str
instruction: str
priority: int = 5 # 1 (highest) to 10 (lowest)
enabled: bool = True
created_at: float = field(default_factory=time.time)
class ProceduralMemory:
"""System prompt rules registry."""
def __init__(self):
self.rules: dict[str, ProceduralRule] = {}
def add_rule(self, description: str, instruction: str, priority: int = 5) -> str:
rule_id = f"rule_{len(self.rules):03d}"
self.rules[rule_id] = ProceduralRule(
rule_id=rule_id, description=description,
instruction=instruction, priority=priority,
)
return rule_id
def build_system_prompt(self, base_prompt: str) -> str:
active = sorted(
[r for r in self.rules.values() if r.enabled],
key=lambda r: r.priority,
)
if not active:
return base_prompt
rules_text = "\n".join(f"- {r.instruction}" for r in active)
return f"{base_prompt}\n\nBehavioral rules:\n{rules_text}"
def list_rules(self) -> list[dict[str, Any]]:
return [{"id": r.rule_id, "description": r.description,
"priority": r.priority, "enabled": r.enabled}
for r in self.rules.values()]
class MemorySystem:
"""Unified interface to all four memory types."""
def __init__(self, user_id: str, max_context_tokens: int = 40000):
self.user_id = user_id
self.in_context = InContextMemory(max_tokens=max_context_tokens)
self.episodic = EpisodicMemory()
self.semantic = SemanticMemory()
self.procedural = ProceduralMemory()
def status(self) -> dict[str, Any]:
return {
"user_id": self.user_id,
"in_context_tokens": self.in_context.token_usage,
"episodic_episodes": self.episodic.count(self.user_id),
"semantic_profile": self.semantic.get_profile(self.user_id).to_context_string(),
"procedural_rules": len(self.procedural.rules),
}
if __name__ == "__main__":
print("Memory Architecture Taxonomy — Lesson 07-01")
print("=" * 50)
ms = MemorySystem(user_id="user_001")
# Demonstrate each memory type
print("\n[In-context] Adding conversation messages:")
ms.in_context.add("user", "I'm working on a FastAPI service for healthcare data.")
ms.in_context.add("assistant", "Great! What specifically are you building?")
ms.in_context.add("user", "A REST API for patient records with JWT auth.")
print(f" Messages: {len(ms.in_context.messages)}")
print(f" Token usage: {ms.in_context.token_usage}")
print("\n[Episodic] Storing past events:")
ep1 = ms.episodic.store(
"User decided to use PostgreSQL for the database after comparing SQLite options.",
user_id="user_001", session_id="sess_001",
tags=["database", "architecture"], importance=4.5,
)
ep2 = ms.episodic.store(
"User prefers async Python (asyncio/FastAPI) over synchronous Django.",
user_id="user_001", session_id="sess_001",
tags=["preferences", "python"], importance=4.0,
)
print(f" Stored episodes: {ep1}, {ep2}")
retrieved = ms.episodic.retrieve("database choice PostgreSQL", user_id="user_001")
print(f" Retrieved for 'database choice': {retrieved[0].text[:60]}...")
print("\n[Semantic] User profile:")
profile = ms.semantic.get_profile("user_001")
profile.update("name", "Alex")
profile.update("expertise_level", "expert")
profile.update("preferred_language", "Python")
profile.update("domain_focus", ["healthcare", "data engineering"])
ms.semantic.set_fact("user_001", "prefers_brevity", True)
ms.semantic.set_fact("user_001", "dislikes_verbose_explanations", True)
print(f" Profile: {profile.to_context_string()}")
print("\n[Procedural] System prompt rules:")
ms.procedural.add_rule("Brevity", "Keep responses concise. Prefer bullets over paragraphs.", priority=2)
ms.procedural.add_rule("Code style", "Default to Python with type hints and async patterns.", priority=3)
ms.procedural.add_rule("Domain", "User works in healthcare — be aware of HIPAA implications.", priority=1)
prompt = ms.procedural.build_system_prompt("You are a helpful coding assistant.")
print(f" Rules added: {len(ms.procedural.rules)}")
print(f" Full system prompt:\n{prompt}")
print("\n[MemorySystem] Status:")
status = ms.status()
for k, v in status.items():
print(f" {k}: {v}")