The Problem
Every modern AI system — GPT-4, Claude, Gemini, Llama — is built on attention. Yet most engineers treat it as a black box. When you can't explain what Q, K, V are, why we divide by √d_k, or how multi-head attention captures different relationship types, you can't:
- Debug why your fine-tuned model ignores certain context
- Understand context window limitations
- Reason about inference cost scaling
- Design efficient architectures for your specific problem
The Concept
What Attention Solves
Before transformers, sequential models (RNNs, LSTMs) had a fundamental problem: to connect the first word in a sentence to the last, information had to pass through every intermediate step, degrading with distance.
Attention gives every token direct access to every other token, regardless of distance.
Why use 8 attention heads instead of 1? What does each head learn differently?
With 8 heads, each projects Q/K/V into different subspaces. One head might learn subject-verb relationships, another might attend to adjectives modifying nouns, another tracks pronoun references. The outputs are concatenated and projected back — the model combines diverse perspectives. Total compute stays roughly constant because each head is d_model/8 dimensional.
Traditional (sequential):
"The cat sat on the mat" → word₁ → word₂ → word₃ → ... → word₆
(Information from "The" degrades by the time we reach "mat")
Attention (parallel):
"The cat sat on the mat"
↕ ↕ ↕ ↕ ↕ ↕ (All tokens see all other tokens directly)
The Core Formula
Attention(Q, K, V) = softmax(Q · K^T / √d_k) · V
Breaking this down:
| Symbol | Name | What it is | Shape |
|---|
| Q | Query | "What am I looking for?" | (seq_len, d_k) |
| K | Key | "What do I contain?" | (seq_len, d_k) |
| V | Value | "What information do I provide?" | (seq_len, d_v) |
| d_k | Key dimension | Scaling factor | scalar |
Step by Step
- Q · K^T — Compute attention scores: how relevant is each key to each query
- / √d_k — Scale down to prevent softmax saturation (large dot products → extreme softmax)
- softmax — Normalize scores to probabilities (each row sums to 1)
- × V — Weight values by their attention scores
Why √d_k?
Without scaling, as d_k grows, the dot products grow in magnitude (variance = d_k). Large values push softmax into regions with near-zero gradients. Dividing by √d_k keeps the variance at 1 regardless of dimension.
Multi-Head Attention
Instead of one attention function, use h parallel heads, each with different learned projections:
head_i = Attention(Q·W_Q_i, K·W_K_i, V·W_V_i)
MultiHead(Q, K, V) = Concat(head_1, ..., head_h) · W_O
Why multiple heads? Different heads learn different relationship types:
- Head 1 might learn syntactic relationships (subject-verb agreement)
- Head 2 might learn positional/proximity patterns
- Head 3 might learn semantic similarity
Self-Attention vs Cross-Attention
| Type | Q from | K, V from | Used in |
|---|
| Self-attention | Same sequence | Same sequence | GPT (decoder), BERT (encoder) |
| Cross-attention | Decoder | Encoder output | Translation, image captioning |
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Sources & Further Reading, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.