Phase 3: LLM Architecture · ~120 minutes · Python

Pre-Training a Mini GPT (124M Parameters)

GPT-2 Small has 124 million parameters. That's 12 transformer layers, 12 attention heads, and 768-dimensional embeddings. You can train it from scratch on a single GPU in a few hours. Most people never do this. They use pre-trained checkpoints. But if you don't train one yourself, you don't actually understand what's happening inside the model you're building products on.

What you will learn

Implement the full GPT-2 architecture (124M parameters) from scratch: token embeddings, positional embeddings, transformer blocks, and the language model head
Train a GPT model on a text corpus using next-token prediction with cross-entropy loss
Implement autoregressive text generation with temperature sampling and top-k/top-p filtering
Monitor training loss curves and validate that the model learns coherent language patterns

Introduction

Type: Build
Languages: Python (with numpy)
Prerequisites: Tokenizers: BPE, WordPiece, SentencePiece, Building a Tokenizer from Scratch, Data Pipelines for Pre-Training
Time: ~120 minutes

Objective

Learning objectives

Implement the full GPT-2 architecture (124M parameters) from scratch: token embeddings, positional embeddings, transformer blocks, and the language model head
Train a GPT model on a text corpus using next-token prediction with cross-entropy loss
Implement autoregressive text generation with temperature sampling and top-k/top-p filtering
Monitor training loss curves and validate that the model learns coherent language patterns

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.