Phase 0: Attention Mechanisms · ~45 minutes · Python
Positional Encoding — Sinusoidal, RoPE, ALiBi
Attention is permutation-invariant. "The cat sat on the mat" and "mat the on sat cat the" produce the same output without positional signal. Three algorithms fix it — each with a different bet on what "position" means.
What you will learn
- Implement positional encoding — sinusoidal, rope, alibi from scratch
- Understand the math and intuition behind the algorithm
- Use production libraries for the same task
- Ship a reusable artifact
Introduction
- Type: Build
- Languages: Python
- Prerequisites: Self-Attention from Scratch, Multi-Head Attention
- Time: ~45 minutes
The Problem
Scaled dot-product attention is order-blind. The attention matrix softmax(Q K^T / √d) V is computed from pairwise similarities. Shuffle the rows of X, get the rows of the output shuffled the same way. Nothing inside attention cares about position.
That is not a bug in a bag-of-words model. For language, code, audio, video — anything where order carries meaning — it is fatal.
The fix is to inject position into the embeddings somehow. Three eras of answers:
- Absolute sinusoidal (Vaswani 2017). Add
sin/cos of position to the embedding. Simple, learnable-free, extrapolates poorly beyond trained lengths. - RoPE — Rotary Position Embeddings (Su 2021). Rotate Q and K vectors by an angle proportional to position. Encodes relative position directly in the dot product. Dominant in 2026.
- ALiBi — Attention with Linear Biases (Press 2022). Skip embeddings entirely; add a per-head linear penalty to attention scores based on distance. Excellent length extrapolation.
As of 2026, essentially every frontier open model uses RoPE: Llama 2/3/4, Qwen 2/3, Mistral, Mixtral, DeepSeek-V3, Kimi. A handful of long-context models use ALiBi or its modern variants. Absolute sinusoidal is historical.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy