Phase 0: Attention Mechanisms · ~90 minutes · Python

Self-Attention from Scratch

Attention is a lookup table where every word asks "who matters to me?" - and learns the answer.

What you will learn

Implement scaled dot-product self-attention from scratch using only NumPy, including query/key/value projections and the softmax-weighted sum
Build a multi-head attention layer that splits heads, computes parallel attention, and concatenates results
Trace how the attention matrix captures token relationships and explain why scaling by sqrt(d_k) prevents softmax saturation
Apply causal masking to convert bidirectional attention into autoregressive (decoder-style) attention

Introduction

Type: Build
Languages: Python
Prerequisites: the Deep Learning from Scratch course, the NLP & Speech Processing course
Time: ~90 minutes

Objective

Learning objectives

Implement scaled dot-product self-attention from scratch using only NumPy, including query/key/value projections and the softmax-weighted sum
Build a multi-head attention layer that splits heads, computes parallel attention, and concatenates results
Trace how the attention matrix captures token relationships and explain why scaling by sqrt(d_k) prevents softmax saturation
Apply causal masking to convert bidirectional attention into autoregressive (decoder-style) attention

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.