The Problem
Recurrent Neural Networks (RNNs) process sequences word by word. For long sentences, information from early words fades by the time we reach the end. "The cat, which was sitting on the mat that was near the window, looked outside." By the time we reach "looked," the subject "cat" is distant.
RNNs also can't parallelize training. Each step waits for the previous. This makes training slow.
The Concept
The Intuition: A Retrieval System Inside the Network
Attention is best understood as a soft, differentiable retrieval system. In a traditional database, you write a query, match it against keys, and retrieve the values of matching records — a hard lookup. Attention does the same thing, but instead of returning one exact match, it returns a weighted average of all values, weighted by how well each key matches the query.
Every token in the sequence generates three vectors by multiplying its embedding with learned projection matrices:
- Query (Q): "What am I looking for?" — this token's question to the rest of the sequence.
- Key (K): "What do I contain?" — this token's label, what it offers to others.
- Value (V): "What information do I provide?" — the actual content this token contributes to the output.
The attention weight between token i (query) and token j (key) is the dot product Q_i · K_j — a similarity score. High similarity means "this key is relevant to my query." The scores are softmax-normalized so they sum to 1, then used to compute a weighted sum of values. This gives every output position a context-aware blend of information from all input positions.
Traditional database: query → match keys → return ONE value (hard lookup)
Attention: query → match ALL keys → return WEIGHTED AVERAGE of values (soft lookup)
The key insight: the weights are learned dynamically based on content, not fixed by position. A pronoun like "it" can attend to "the cat" because their query-key projections match — the model learned that pronouns should look for their antecedents. This content-based routing is what makes attention so powerful: the model learns where to look based on what it needs.
In scaled dot-product attention, why do we divide Q·Kᵀ by √d_k before softmax?
With dimension d_k, dot products have variance proportional to d_k. For d_k=512, raw dot products can reach ±100+, pushing softmax into saturated regions where gradients vanish. Dividing by √d_k keeps variance ≈ 1, so softmax stays in a healthy range with usable gradients.
Traditional RNN: h_t = f(h_{t-1}, x_t) (sequential bottleneck)
Self-Attention: For each position i:
Query q_i = W_q · x_i
For each position j:
Score(i,j) = q_i · (W_k · x_j) (compatibility)
Weight(i,j) = softmax_j(Score)
Output_i = Σ_j Weight(i,j) · (W_v · x_j)
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing, Sources & Further Reading — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.