Phase 4: Training LLMs · ~60 minutes · Python
Native Sparse Attention (DeepSeek NSA)
At 64k tokens, attention eats 70-80% of decode latency. Every open-model lab has a plan to fix it. DeepSeek's NSA (ACL 2025 best paper) is the one that stuck: three parallel attention branches — compressed coarse-grained tokens, selectively retained fine-grained tokens, and sliding windows for local context — combined through a learned gate. It is hardware-aligned (kernel-friendly), natively trainable (works in pre-training, not bolted on at inference), and on 64k decodes it runs faster than FlashAttention while matching or beating full attention quality. This lesson builds the three branches end-to-end and shows why the sparsity is end-to-end differentiable.
Hiring signal: Understanding of native sparse attention (deepseek nsa) internals
What you will learn
- State the three NSA attention branches and what each one captures.
- Explain why NSA is "natively trainable" where prior sparse-attention methods were inference-only.
- Compute the attention compute savings of NSA versus full attention at 64k context as a function of compression block size and selection top-k.
- Implement the three-branch combination in stdlib Python on a short synthetic sequence and verify the gating weights behave.
Introduction
Type: Build Languages: Python (stdlib) Prerequisites: Phase 7 · 12 (KV cache, flash-attention), Phase 7 · 15 (attention variants), Phase 10 · 16 (differential attention) Time: ~60 minutes
Learning Objectives
- State the three NSA attention branches and what each one captures.
- Explain why NSA is "natively trainable" where prior sparse-attention methods were inference-only.
- Compute the attention compute savings of NSA versus full attention at 64k context as a function of compression block size and selection top-k.
- Implement the three-branch combination in stdlib Python on a short synthetic sequence and verify the gating weights behave.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build It, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy