The Concept
Discriminative vs. Generative: Two Ways to Model Data
A discriminative model learns P(y|x) — the probability of the label given the input. It cares about the boundary between classes, not the data itself. A logistic regression doesn't know what a "spam email" looks like — it only knows which features push the probability toward "spam."
A generative model learns P(x) — the probability distribution of the data itself (or P(x,y) jointly). It knows what a spam email looks like, what a non-spam email looks like, and can generate new examples of either. The generative model has a richer understanding: it knows the full data distribution, not just the decision boundary.
Discriminative: learn P(y|x) — "given this input, what's the label?"
Generative: learn P(x) — "what does the data look like?"
or P(x,y) — "what does each class look like?"
Autoencoders: Learning to Compress
An autoencoder is the simplest generative model. It has two parts: an encoder that compresses the input into a low-dimensional "latent code" z, and a decoder that reconstructs the input from z. The network is trained to minimize reconstruction error — the difference between the input and the reconstruction.
The key insight: by forcing the data through a bottleneck (the latent code is much smaller than the input), the encoder must learn the most important features and discard noise. The latent code is a compressed representation that captures the essential structure of the data.
Input x → [Encoder] → latent z (low-dim) → [Decoder] → Reconstruction x'
Loss = ||x - x'||² (minimize reconstruction error)
Autoencoders are useful for dimensionality reduction (like a nonlinear PCA), denoising (train on noisy inputs, reconstruct clean outputs), and anomaly detection (anomalies will have high reconstruction error because they don't match the learned distribution). But a vanilla autoencoder can't generate new data — it can only reconstruct inputs it has seen.
Variational Autoencoders (VAEs): Learning to Generate
A Variational Autoencoder adds a probabilistic twist: instead of encoding an input as a single point z, the encoder outputs a distribution over z (a mean and variance). The decoder then samples from this distribution to reconstruct the input.
This seemingly small change has a profound effect: because the latent space is now a continuous probability distribution, you can sample new points from it and decode them into novel data. Sample a random z from the learned latent distribution, pass it through the decoder, and you get a new data sample that resembles the training data but isn't a copy of any specific example.
VAEs train with a dual loss: reconstruction loss (the data should be reconstructable) plus KL divergence (the latent distribution should be close to a standard normal, ensuring the latent space is smooth and well-organized). The KL term is what makes the latent space generative — without it, the encoder could map each input to a unique point with no structure, and sampling would produce garbage.
VAE loss = Reconstruction loss + KL divergence
= ||x - x'||² + KL(q(z|x) || N(0,1))
The intuition for the KL term: it pushes the encoded distributions to overlap and fill the latent space like a smooth cloud, rather than clustering as isolated points. This smoothness is what allows interpolation — you can smoothly transition from one sample to another by moving through the latent space.
Generative Adversarial Networks (GANs): Learning by Competition
A GAN takes a completely different approach to generation. Instead of modeling the data distribution explicitly, it sets up a game between two networks:
- Generator: takes random noise
z and produces fake data. Its goal: fool the discriminator. - Discriminator: takes a data sample (real or fake) and predicts whether it's real or generated. Its goal: correctly distinguish real from fake.
They train in alternation: the discriminator gets better at spotting fakes, so the generator must get better at producing convincing fakes, so the discriminator must get even better, and so on. The equilibrium (if training is stable) is a generator that produces data indistinguishable from real data.
Generator: G(z) → fake data, tries to fool D
Discriminator: D(x) → P(real), tries to detect fakes
Loss: min_G max_D [ E[log D(x)] + E[log(1 - D(G(z)))] ]
GANs produce sharper, more realistic samples than VAEs (which tend to produce blurry outputs due to the reconstruction loss). But GANs are notoriously hard to train — the generator and discriminator can become unbalanced (one dominates the other), training can oscillate or collapse (the generator finds one output that fools the discriminator and produces nothing else — "mode collapse").
You train a GAN and notice the generator produces the same image regardless of the input noise. What's happening and what's a common fix?
Mode collapse is the most common GAN failure: the generator finds a single output that the discriminator can't distinguish from real, and produces it for every input. Fixes include minibatch discrimination (D looks at batches, not individual samples), feature matching (G matches intermediate D features, not just the final output), and Wasserstein GAN loss (provides smoother gradients and reduces mode collapse).
When to Use What
Autoencoder : compression, denoising, anomaly detection (high reconstruction error = anomaly)
VAE : generation with smooth interpolation, data augmentation, learning structured latent spaces
GAN : high-quality image generation, style transfer (when sharp, realistic outputs matter)