Phase 3: LLM Architecture · ~90 minutes · Python

Tokenizers: BPE, WordPiece, SentencePiece

Your LLM does not read English. It reads integers. The tokenizer decides whether those integers carry meaning or waste it.

What you will learn

Implement BPE, WordPiece, and Unigram tokenization algorithms from scratch and compare their merge strategies
Explain how vocabulary size affects model efficiency: too small creates long sequences, too large wastes embedding parameters
Analyze tokenization artifacts across languages and code, identifying where specific tokenizers break down
Use the tiktoken and sentencepiece libraries to tokenize text and inspect the resulting token IDs

Introduction

Type: Build
Languages: Python
Prerequisites: Earlier lessons in this course
Time: ~90 minutes

Objective

Learning objectives

Implement BPE, WordPiece, and Unigram tokenization algorithms from scratch and compare their merge strategies
Explain how vocabulary size affects model efficiency: too small creates long sequences, too large wastes embedding parameters
Analyze tokenization artifacts across languages and code, identifying where specific tokenizers break down
Use the tiktoken and sentencepiece libraries to tokenize text and inspect the resulting token IDs

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.