The Problem
Word2Vec left two open questions.
First, there was a parallel line of research that factorized the co-occurrence matrix directly (LSA, HAL) rather than doing online skip-gram updates. Was Word2Vec's iterative approach fundamentally better, or was the difference an artifact of how the two methods handled counts? GloVe answered that: matrix factorization with a thoughtfully chosen loss matches or beats Word2Vec, and costs less to train.
Second, neither method had a story for words it had never seen. Zoomer-approved, dogecoin, any proper noun coined last week, every inflected form of a rare root. FastText fixed this by embedding character n-grams: a word is the sum of its parts, including morphemes, so even out-of-vocabulary words get a sensible vector.
Third, once transformers arrived, the question shifted again. Word-level vocabularies cap out around a million entries; real language is more open than that. Byte-pair encoding (BPE) and its relatives solved this by learning a vocabulary of frequent subword units that covers everything. Every modern tokenizer for every modern LLM is a subword tokenizer.
This lesson walks all three, then explains which to reach for when.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.