The Problem
A user has 200 past conversations with your AI assistant. Their preferences, past decisions, domain expertise, and ongoing projects are all in there. You can't fit all of that in a context window — even Claude's 200k token limit won't accommodate 200 full conversations.
The solution is a vector store: a database that stores memories as semantic embeddings, then retrieves only the ones relevant to the current context. Instead of loading all memories into every prompt, you load the top 5 most relevant ones — typically under 1,000 tokens.
This is external memory: memory that persists across sessions, scales to millions of records, and retrieves by meaning rather than exact match. It's what separates an agent that "remembers nothing" from one that remembers what matters.
The engineering challenge is not just storing memories — it's retrieving the right ones. Pure semantic similarity is not enough: a memory from 18 months ago that's topically relevant but factually outdated should not rank above a memory from last week. Combined scoring is the key.
Memory record schema
Each memory is a structured document: memory_id, text, embedding, timestamp, user_id, session_id, tags, importance_score, and status (active/superseded/archived). The status field is the critical addition — it lets you mark outdated memories without deleting them, preserving an audit trail while preventing stale information from influencing retrieval.
Memory as a vector store
The architecture: one vector collection per user (or one shared collection with user_id metadata filters). Each memory is an embedded document. At the start of each session, search for memories relevant to the current context and inject the top-5 into the system prompt.
Storage: ChromaDB (local development) or pgvector (production Postgres). Both support cosine similarity search. ChromaDB is simpler to get started; pgvector co-locates memory with your application database and scales to billions of vectors.
Memory schema:
@dataclass
class Memory:
memory_id: str # UUID
text: str # self-contained statement
user_id: str # isolation between users
session_id: str # provenance
timestamp: float # Unix time — used for recency scoring
tags: list[str] # for filtered retrieval
importance_score: float # 1.0 to 5.0 — set at ingestion
embedding: list[float] # computed at store time
status: str # active | superseded | archived
Injection: retrieved memories are prepended to the system prompt as plain text:
Relevant past context:
1. [3d ago, importance 4.5] User uses PostgreSQL with pgvector for production vector storage.
2. [14d ago, importance 4.0] User prefers concise responses without lengthy explanations.
3. [30d ago, importance 3.5] User is building a patient record retrieval system for a healthcare startup.
The model reads this context naturally and uses it without any special prompting.
Combined relevance scoring
Two factors determine retrieval rank:
Semantic relevance: cosine similarity between the query embedding and each memory embedding. Measures topical relevance — is this memory about the same subject as the current query?
Recency: exp(-days_since * decay_rate). Memories from last week get a high recency score; memories from last year get a low one. decay_rate = 0.05 gives memories a half-life of about 14 days.
Combined score: 0.7 * semantic_score + 0.3 * recency_score
The 70/30 split prioritizes relevance but gives meaningful weight to recency. A highly relevant memory from 2 years ago (semantic=0.9, recency=0.05) scores 0.65. A moderately relevant memory from last week (semantic=0.6, recency=0.85) scores 0.68 — slightly higher. This is the intended behavior: recent information about the same topic edges out older information.
Why combine semantic relevance AND recency in memory retrieval scoring?
Explanation: User context evolves. A memory from 18 months ago "user prefers Slack" may be outdated — they switched to Teams. Pure semantic relevance retrieves the most similar memory, which may be the oldest and most stale. Recency weighting ensures recent memories rank higher. Combined with the status system (superseding outdated memories), this creates a retrieval layer that naturally prioritizes current, accurate information.
Memory ingestion pipeline
The ingestion pipeline runs after each conversation ends. It extracts key memories using an LLM, deduplicates against existing memories, and stores the new ones.
Step 1 — Extract: pass the conversation to claude-haiku-4-5 with a structured extraction prompt. The model returns 3–5 self-contained statements: "User uses PostgreSQL with pgvector in production", "User prefers Python over Java", "User's project requires HIPAA compliance in AWS us-east-1".
Self-contained is critical: a memory retrieved 3 months from now needs to make sense without the surrounding conversation. "User agreed" is not a valid memory. "User agreed to use FastAPI for the REST layer instead of Flask" is.
Step 2 — Embed: compute the embedding for each extracted memory.
Step 3 — Deduplicate: compute cosine similarity between the new memory and all existing memories for this user. If similarity > 0.95, update the existing memory's importance score rather than creating a duplicate. This prevents the store from growing with redundant records.
Step 4 — Store: write to the vector store with full metadata.
Run asynchronously: memory extraction and embedding takes 1–5 seconds. Running synchronously would delay the end of the user's session. The next session doesn't need memories from the current session immediately. Use a background task or job queue.
The memory ingestion pipeline runs after each conversation. Why asynchronously?
Explanation: The user's session has just ended — they don't need to wait for memory processing. Async ingestion means the conversation ends, a background job starts extraction, and the user can start a new session while memories are being processed. The new memories become available within seconds to minutes, invisible to the user. Synchronous ingestion would add a 2-5 second delay to every conversation ending — a poor UX trade for no user-visible benefit.
Build
Build it
Real embeddings need a model, but the retrieval logic — score by word overlap, rank, return top-K — is easy to see with a tiny stand-in "embedding" based on shared keywords.
def keyword_overlap_score(query, memory_text):
query_words = set(query.lower().split())
memory_words = set(memory_text.lower().split())
if not query_words:
return 0.0
return len(query_words & memory_words) / len(query_words)
memories = [
"User uses PostgreSQL with pgvector for production vector storage.",
"User prefers concise responses without lengthy explanations.",
"User is building a patient record retrieval system for healthcare.",
"User switched from Java to Python for backend development.",
"User likes bullet points over long paragraphs.",
]
def retrieve(query, memories, top_k=2):
scored = [(keyword_overlap_score(query, m), m) for m in memories]
scored.sort(key=lambda pair: pair[0], reverse=True)
return scored[:top_k]
query = "What database does the user use for vector storage?"
results = retrieve(query, memories, top_k=2)
print(f"Query: {query}\n")
print("Top retrieved memories:")
for score, text in results:
print(f" score={score:.2f} {text}")
The code artifact lesson_07_03_vector_memory.py implements the full pipeline:
VectorMemoryStore — wraps ChromaDB (or an in-memory fallback) with store(), retrieve(), find_near_duplicates(), and update_status() methodsMemoryIngestionPipeline — LLM extraction using claude-haiku-4-5, deduplication via cosine similarity threshold, async-ready store callMemoryRetriever — combined semantic + recency scoring, format_for_prompt() helper that formats retrieved memories as natural prose for system prompt injectionMemoryAwareAgent — builds a personalized system prompt by retrieving relevant memories before each turn, calls end_session_and_ingest() to persist new memories after the session
The demo simulates 5 sessions, ingests memories from each, then starts a 6th session and demonstrates cross-session recall. It also shows contradiction handling: a new statement ("I switched from Java to Python") supersedes the old memory and takes the top retrieval slot.