Phase 3: Deep Learning Core · 55 min · PyTorch

RNNs & Sequence Models

Before attention, sequences meant recurrence. Knowing why RNNs struggled explains why transformers won.

Hiring signal: Understands sequence modeling tradeoffs and the limits that motivated attention

What you will learn

Explain how an RNN processes sequences via a hidden state
Describe vanishing/exploding gradients and why they limit RNNs
Contrast LSTM/GRU gating with a vanilla RNN
Build an LSTM for sequence classification or forecasting in PyTorch
Articulate why attention/transformers superseded recurrence

The Problem

Text, time series, audio, and sensor logs are sequences — order carries meaning, and "I did not like it" is the opposite of "I did like it." A plain feed-forward net has no notion of order or memory. Recurrent neural networks were the answer for a decade, and while transformers now dominate NLP, RNNs/LSTMs are still used for streaming and small-scale time series — and, more importantly for interviews, understanding why RNNs struggled is the cleanest way to explain why attention exists. Skipping RNNs leaves a hole in your transformer story.

The Concept

An RNN reads a sequence one step at a time, carrying a hidden state that summarizes everything seen so far:

h_t = f(W_x · x_t + W_h · h_{t-1} + b)     # same weights reused at every step
y_t = g(W_y · h_t)

Hidden state h_t : the network's "memory" of the past
Weight sharing   : one cell applied across all timesteps (like conv across space)
Sequential       : step t depends on step t-1 -> CANNOT parallelize over time

Why can't you parallelize an RNN across time steps during training, but a Transformer can be parallelized?

An RNN computes h_t = f(h_{t-1}, x_t) — you can't compute h_t until h_{t-1} is done. A Transformer's self-attention computes all positions at once: Q, K, V for the entire sequence are computed in parallel. This is why Transformers train much faster on GPUs.

That last point is the seed of the transformer's victory.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Transformer