The Concept
The Four Components and Why Each Is Necessary
Attention alone is not enough to build a useful model. A transformer block wraps attention with three more components, each solving a specific problem. Understanding why each component exists is the key to answering transformer interview questions:
Multi-Head Self-Attention mixes information across positions. This is the only component that lets tokens interact with each other. Without it, each token would be processed in isolation, and the model couldn't understand context, coreference, or relationships between words.
Feed-Forward Network (FFN) processes each position independently with a non-linearity. After attention has mixed information, each token needs to think about what it just learned — transform it, filter it, extract features. The FFN is a two-layer MLP applied to each position separately: it expands the dimensionality (typically 4x), applies a non-linearity (ReLU or GELU), and projects back. This per-position processing is where the model's "reasoning" happens — attention gathers context, the FFN processes it.
Residual connections (x = x + sublayer(x)) solve the vanishing gradient problem in deep networks. In a 96-layer transformer (like GPT-3), backpropagation must pass through 96 blocks. Without residual connections, the gradient gets multiplied by 96 Jacobians and shrinks to zero — early layers never learn. Residuals add a +1 term to the gradient path, providing a direct highway for signal to flow from the output all the way back to the first layer. This is the same insight as ResNets in computer vision.
Layer normalization stabilizes activations by normalizing them to zero mean and unit variance within each layer. Without it, activations can grow or shrink exponentially through deep stacks, causing training to diverge. LayerNorm keeps everything in a reasonable range, allowing higher learning rates and stable training.
x ──► [ Multi-Head Self-Attention ] ──► (+) ──► [ LayerNorm ] ──► h
│ ▲
└────────────── residual ────────────────┘
h ──► [ Feed-Forward Network ] ──► (+) ──► [ LayerNorm ] ──► out
│ ▲
└────────────── residual ───────────┘
Four jobs, four components:
Attention : mix information across positions (the only cross-token step)
Feed-Forward : process each position independently with a non-linearity
Residual : add input back to output so gradients flow through deep stacks
LayerNorm : stabilize activations so training does not blow up
Transformer blocks use residual connections: x = x + sublayer(x). Why are these critical for deep transformers (e.g., 96 layers in GPT-3)?
In a 96-layer transformer, backprop must pass through 96 blocks. Without residual connections, the gradient gets multiplied by 96 Jacobians — it vanishes. Residuals add a +1 term to the gradient path, so signal flows directly from output to every layer. This is why deep transformers (and ResNets) can train at all.
Attention : mix information across positions (the only cross-token step)
Feed-Forward : process each position independently with a non-linearity
Residual : add input back to output so gradients flow through deep stacks
LayerNorm : stabilize activations so training does not blow up
Pre-norm vs post-norm
The original paper applied LayerNorm after the residual add (post-norm). Modern LLMs apply it before the sub-layer (pre-norm) because it trains far more stably at depth:
Post-norm (2017): out = LayerNorm(x + Sublayer(x))
Pre-norm (modern): out = x + Sublayer(LayerNorm(x)) <- gradients flow cleanly
Three architecture families
Encoder-only (BERT) : bidirectional attention. Best for understanding/classification.
Decoder-only (GPT) : causal (masked) attention. Best for generation. Powers most LLMs.
Encoder-decoder (T5) : encoder reads input, decoder generates. Best for translation/seq2seq.