Phase 4: Training LLMs · ~60 minutes · Python

Native Sparse Attention (DeepSeek NSA)

At 64k tokens, attention eats 70-80% of decode latency. Every open-model lab has a plan to fix it. DeepSeek's NSA (ACL 2025 best paper) is the one that stuck: three parallel attention branches — compressed coarse-grained tokens, selectively retained fine-grained tokens, and sliding windows for local context — combined through a learned gate. It is hardware-aligned (kernel-friendly), natively trainable (works in pre-training, not bolted on at inference), and on 64k decodes it runs faster than FlashAttention while matching or beating full attention quality. This lesson builds the three branches end-to-end and shows why the sparsity is end-to-end differentiable.

What you will learn

State the three NSA attention branches and what each one captures.
Explain why NSA is "natively trainable" where prior sparse-attention methods were inference-only.
Compute the attention compute savings of NSA versus full attention at 64k context as a function of compression block size and selection top-k.
Implement the three-branch combination in stdlib Python on a short synthetic sequence and verify the gating weights behave.

Introduction

Type: Build
Languages: Python (stdlib)
Prerequisites: KV Cache, Flash Attention & Inference Optimization, Attention Variants — Sliding Window, Sparse, Differential, Differential Attention (V2)
Time: ~60 minutes

Objective

Learning objectives

State the three NSA attention branches and what each one captures.
Explain why NSA is "natively trainable" where prior sparse-attention methods were inference-only.
Compute the attention compute savings of NSA versus full attention at 64k context as a function of compression block size and selection top-k.
Implement the three-branch combination in stdlib Python on a short synthetic sequence and verify the gating weights behave.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.

Key terms in this lesson

Inference