Phase 7: Agent Memory Systems · 75 min · Python · Anthropic SDK · ChromaDB

External Vector Store Memory

A vector store is the long-term memory that holds everything — and retrieves only the relevant pieces when needed.

Hiring signal: Persistent memory is the feature that makes AI assistants feel like relationships rather than transactions. Products like Notion AI, Cursor, and enterprise assistants all implement external memory. This is among the most requested features in enterprise AI deployments and the skill that makes agents feel genuinely intelligent.

What you will learn

Store and retrieve episodic memories in a vector database using combined semantic + recency scoring
Build a memory ingestion pipeline: conversation -> extract key moments -> embed -> store with deduplication
Implement contradiction handling: supersede outdated memories rather than deleting them

The Problem

A user has 200 past conversations with your AI assistant. Their preferences, past decisions, domain expertise, and ongoing projects are all in there. You can't fit all of that in a context window — even Claude's 200k token limit won't accommodate 200 full conversations.

The solution is a vector store: a database that stores memories as semantic embeddings, then retrieves only the ones relevant to the current context. Instead of loading all memories into every prompt, you load the top 5 most relevant ones — typically under 1,000 tokens.

This is external memory: memory that persists across sessions, scales to millions of records, and retrieves by meaning rather than exact match. It's what separates an agent that "remembers nothing" from one that remembers what matters.

The engineering challenge is not just storing memories — it's retrieving the right ones. Pure semantic similarity is not enough: a memory from 18 months ago that's topically relevant but factually outdated should not rank above a memory from last week. Combined scoring is the key.

Memory record schema
Each memory is a structured document: memory_id, text, embedding, timestamp, user_id, session_id, tags, importance_score, and status (active/superseded/archived). The status field is the critical addition — it lets you mark outdated memories without deleting them, preserving an audit trail while preventing stale information from influencing retrieval.

Memory as a Vector Store

The architecture: one vector collection per user (or one shared collection with user_id metadata filters). Each memory is an embedded document. At the start of each session, search for memories relevant to the current context and inject the top-5 into the system prompt.

Storage: ChromaDB (local development) or pgvector (production Postgres). Both support cosine similarity search. ChromaDB is simpler to get started; pgvector co-locates memory with your application database and scales to billions of vectors.

Memory schema:

@dataclass
class Memory:
    memory_id: str          # UUID
    text: str               # self-contained statement
    user_id: str            # isolation between users
    session_id: str         # provenance
    timestamp: float        # Unix time — used for recency scoring
    tags: list[str]         # for filtered retrieval
    importance_score: float # 1.0 to 5.0 — set at ingestion
    embedding: list[float]  # computed at store time
    status: str             # active | superseded | archived

Injection: retrieved memories are prepended to the system prompt as plain text:

Relevant past context:
  1. [3d ago, importance 4.5] User uses PostgreSQL with pgvector for production vector storage.
  2. [14d ago, importance 4.0] User prefers concise responses without lengthy explanations.
  3. [30d ago, importance 3.5] User is building a patient record retrieval system for a healthcare startup.

The model reads this context naturally and uses it without any special prompting.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Combined Relevance Scoring, Memory Ingestion Pipeline, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.