Phase 3: LLM Architecture · ~90 minutes · Python

Building a Tokenizer from Scratch

Lesson 01 gave you a toy. This lesson gives you a weapon.

What you will learn

Build a production-grade BPE tokenizer that handles Unicode, whitespace normalization, and special tokens
Implement byte-level fallback so the tokenizer can encode any input (including emoji, CJK, and code) without unknown tokens
Add pre-tokenization regex patterns that split text at word boundaries before applying BPE merges
Train a custom tokenizer on a corpus and evaluate its compression ratio against tiktoken on multilingual text

Introduction

Type: Build
Languages: Python
Prerequisites: Tokenizers: BPE, WordPiece, SentencePiece
Time: ~90 minutes

Objective

Learning objectives

Build a production-grade BPE tokenizer that handles Unicode, whitespace normalization, and special tokens
Implement byte-level fallback so the tokenizer can encode any input (including emoji, CJK, and code) without unknown tokens
Add pre-tokenization regex patterns that split text at word boundaries before applying BPE merges
Train a custom tokenizer on a corpus and evaluate its compression ratio against tiktoken on multilingual text

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.

Key terms in this lesson

Token