The Problem
TF-IDF and Word2Vec produced flat vectors that ignored word order. A classifier built on them could not tell dog bites man from man bites dog. Word order sometimes carries the signal.
Two families of architectures filled that gap before transformers arrived.
Convolutional nets for text (TextCNN). Apply 1D convolutions over sequences of word embeddings. A filter of width 3 is a learnable trigram detector: it spans three words and outputs a score. Stack different widths (2, 3, 4, 5) to detect multi-scale patterns. Max-pool to a fixed-size representation. Flat, parallel, fast.
Recurrent nets (RNN, LSTM, GRU). Process tokens one at a time, maintaining a hidden state that carries information forward. Sequential, memory-bearing, flexible input lengths. Dominated sequence modeling from 2014 to 2017, then attention happened.
This lesson builds both, then names the failure that motivated attention.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.