Phase 0: NLP Foundations · ~45 minutes · Python

GloVe, FastText, and Subword Embeddings

Word2Vec trained one embedding per word. GloVe factorized the co-occurrence matrix. FastText embedded the pieces. BPE bridged to transformers.

Hiring signal: Understanding of glove, fasttext, and subword embeddings internals

What you will learn

Implement glove, fasttext, and subword embeddings from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Build Languages: Python Prerequisites: Phase 5 · 03 (Word2Vec from Scratch) Time: ~45 minutes

The Problem

Word2Vec left two open questions.

First, there was a parallel line of research that factorized the co-occurrence matrix directly (LSA, HAL) rather than doing online skip-gram updates. Was Word2Vec's iterative approach fundamentally better, or was the difference an artifact of how the two methods handled counts? GloVe answered that: matrix factorization with a thoughtfully chosen loss matches or beats Word2Vec, and costs less to train.

Second, neither method had a story for words it had never seen. Zoomer-approved, dogecoin, any proper noun coined last week, every inflected form of a rare root. FastText fixed this by embedding character n-grams: a word is the sum of its parts, including morphemes, so even out-of-vocabulary words get a sensible vector.

Third, once transformers arrived, the question shifted again. Word-level vocabularies cap out around a million entries; real language is more open than that. Byte-pair encoding (BPE) and its relatives solved this by learning a vocabulary of frequent subword units that covers everything. Every modern tokenizer for every modern LLM is a subword tokenizer.

This lesson walks all three, then explains which to reach for when.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build It, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Token