Phase 2: Pre-training & Fine-tuning · ~60 minutes · Python

Attention Variants — Sliding Window, Sparse, Differential

Full attention is a circle. Every token sees every token, and memory pays the price. Four variants bend the shape of the circle and recover half the cost.

Hiring signal: Understanding of attention variants — sliding window, sparse, differential internals

What you will learn

Implement attention variants — sliding window, sparse, differential from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Build Languages: Python Prerequisites: Phase 7 · 02 (Self-Attention), Phase 7 · 03 (Multi-Head), Phase 7 · 12 (KV Cache / Flash Attention) Time: ~60 minutes

The Problem

Full attention costs O(N²) memory and O(N²) compute in sequence length. For a 128K-context Llama 3 70B that is 16 billion attention entries per layer, times 80 layers. Flash Attention (Lesson 12) hides the O(N²) activation memory but does not change the arithmetic cost — every token still attends to every other token.

Three classes of variants change the topology of the attention matrix itself:

Sliding window attention (SWA). Each token attends to a fixed window of neighbors, not the full prefix. Memory and compute drop to O(N · W) where W is the window. Gemma 2/3, Mistral 7B's first layers, Phi-3-Long.
Sparse / block attention. Only selected pairs (i, j) get scored; the rest are forced to zero weight. Longformer, BigBird, OpenAI sparse transformer.
Differential attention. Compute two attention maps with separate Q/K projections, subtract one from the other. Kills the "attention sink" that bleeds weight into the first few tokens. Microsoft's DIFF Transformer (2024).

These coexist. A 2026 frontier model often mixes them: most layers are SWA-1024, every fifth is global full attention, and a handful are differential heads that clean up retrieval. Gemma 3's 5:1 SWA-to-global ratio is the current textbook default.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build It, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Token
Transformer