Phase 6: RAG Systems & Knowledge Grounding · 60 min · sentence-transformers · Chroma · FAISS

Embeddings & Vector Stores

A RAG system is only as good as its retrieval. Retrieval starts with the right embedding and the right vector store.

Hiring signal: Embedding selection and vector store architecture

The Problem

Chunking gives you pieces of text. To find the right piece at query time, you need a representation that captures meaning. Embeddings turn text into dense vectors, and vector stores index those vectors so you can search by semantic similarity. Pick the wrong embedding model or the wrong store, and your RAG system retrieves garbage quickly.

The Concept

What Embeddings Actually Are

An embedding is a numerical representation of meaning. The embedding model takes a piece of text — a sentence, a paragraph, a chunk — and maps it to a fixed-length vector of real numbers (typically 384, 768, or 1536 dimensions depending on the model). The key property is that texts with similar meanings end up close together in this vector space, while texts with different meanings are far apart.

This is not magic — it's a learned mapping. The embedding model is trained on millions of text pairs (question-answer, paraphrase, similar/dissimilar) so that it learns to position vectors in a way that reflects semantic relationships. "How do I set a learning rate?" and "What value should I use for the learning rate?" will have nearly identical vectors because they mean the same thing, even though they share few words. "The learning rate controls step size" will be nearby because it's semantically related. "I like pizza" will be far away.

How Vector Search Works

Once you have embeddings, you need a way to find the closest ones quickly. This is what a vector store (or vector database) does. It indexes all your document embeddings and, given a query embedding, finds the nearest neighbors — the documents whose vectors are closest to the query vector.

Query: "How do I set a learning rate?"
        ↓ embedding model
Vector: [0.02, -0.11, 0.88, ..., 0.04]
        ↓ vector search
Top chunks: "The learning rate controls step size..."
            "Gradient descent updates weights..."
            "Too large a learning rate causes NaN loss..."

The similarity between two vectors is typically measured by cosine similarity (the angle between them) or dot product (which combines angle and magnitude). Cosine similarity is preferred for text embeddings because it's magnitude-invariant — it cares about direction, not length.

The naive approach — compute the distance between the query and every document — works for small corpora but is O(n) and too slow for production. Vector stores use Approximate Nearest Neighbor (ANN) algorithms like HNSW (Hierarchical Navigable Small World) or IVF (Inverted File Index) to find near-neighbors in O(log n) time, trading a small amount of accuracy for a massive speed improvement. For 1 million documents, a brute-force search might take 500ms; HNSW takes <5ms with 99%+ recall.

Choosing the Right Embedding Model

The embedding model determines the quality of your retrieval. A bad embedding model means even the best vector store will retrieve the wrong documents. Key tradeoffs:

Quality: Larger models (OpenAI text-embedding-3-large, 3072 dims) capture more nuance but cost more and produce larger vectors (more storage, slower search).
Speed: Smaller models (all-MiniLM-L6-v2, 384 dims) are fast and free but may miss subtle semantic relationships.
Cost: Hosted models (OpenAI, Cohere) charge per token. Open-source models (sentence-transformers) are free but require GPU/CPU to run.
Domain: General-purpose embeddings may underperform on specialized domains (medical, legal, code). Domain-specific embedding models exist but are less common.

You have a small internal knowledge base (10k documents) and need low-latency retrieval. Which setup is usually best?

For a small, internal corpus, a small open-source embedding model plus a local vector store is cheap, fast, and keeps data on-premise. Large cloud embedders add latency and cost you don't need.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Common Pitfalls, Production Checklist, Interview Framing, Evaluation, Exercises, Key Terms — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.