Phase 0: LLM Fast Track · 50 min · OpenAI SDK · Sentence Transformers · NumPy
Embeddings & Semantic Search
Embeddings are the spatial representation of meaning — the foundational primitive underneath every RAG system, semantic router, and memory store.
Hiring signal: Embeddings underpin every production RAG system. 'Walk me through building a document search system from scratch' is one of the most common AI engineer interview questions.
What you will learn
- Generate embeddings using OpenAI and Sentence Transformers
- Implement cosine similarity search over a collection of documents
- Choose the right embedding model for accuracy vs. speed vs. cost trade-offs
The Problem
A keyword search for "affordable car" misses a document about "budget automobile." An embedding-based search finds it instantly because it understands that "affordable" and "budget" mean the same thing in this context, and "car" and "automobile" are synonyms.
Keyword search matches strings. Semantic search matches meaning. Everything we'll build in the later phases of this course — RAG systems, agent memory, semantic routers — runs on embeddings. This lesson gives you that foundation.
What you'll build
An EmbeddingStore class that can embed documents, store them efficiently in numpy, and retrieve the most semantically similar ones at query time. You'll also benchmark it against keyword search.
What Embeddings Are
Text → fixed-length vector. Similar meaning → similar direction in vector space.
When you pass "The cat sat on the mat" to an embedding model, it returns a vector of ~1536 numbers (for OpenAI) or ~384 numbers (for local models). These numbers encode the semantic content of the text. The key property:
Texts with similar meaning produce vectors that point in similar directions.
The classic demonstration:
embedding("king") - embedding("man") + embedding("woman") ≈ embedding("queen")
This is not magic — it's the model having learned to place semantically related concepts nearby in vector space during training.
Why This Enables Semantic Search
Instead of asking "does this document contain the word 'affordable'?", we ask "is this document's meaning close to the query's meaning?" — measured by the angle between their vectors.
The pipeline:
Query text → embedding model → query vector
↓
compare angle with every document vector
↓
return documents with smallest angle (highest similarity)
Dimensions matter for cost and performance
OpenAI text-embedding-3-small: 1536 dimensions, ~$0.02/1M tokens, best accuracy/cost balance OpenAI text-embedding-3-large: 3072 dimensions, ~$0.13/1M tokens, highest accuracy all-MiniLM-L6-v2 (Sentence Transformers): 384 dimensions, free, offline, fast
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Generating and Comparing Embeddings, Vector Databases: When You Need Them, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy