The Problem
You need to predict house prices. You have data: square footage, bedrooms, location. You could use complex neural networks, but you don't have much data. You need something that works, is interpretable, and trains fast.
Linear regression is the foundation. It models the relationship between input features and a continuous target as a weighted sum. The weights tell you how much each feature influences the prediction.
The Concept
The Intuition: Finding the Best Line
Linear regression is the simplest model that learns from data. The idea is intuitive: you have a scatter plot of data points, and you want to draw a line through them that best captures the trend. "Best" means the line that minimizes the total distance between itself and all the data points.
When you have one input feature (like square footage), the model is a line: y = wx + b. The weight w is the slope — how much the price changes per square foot. The bias b is the intercept — the baseline price at zero square feet. When you have multiple features, the line becomes a plane (or hyperplane in higher dimensions), but the principle is the same: a weighted sum of inputs plus a baseline.
y = w₁x₁ + w₂x₂ + ... + wₙxₙ + b
Where:
y = predicted value (continuous)
xᵢ = input features
wᵢ = learned weights (slopes)
b = bias term (intercept)
Why Mean Squared Error?
The loss function for linear regression is Mean Squared Error (MSE) — the average of the squared differences between predictions and actual values. But why squared? Why not absolute differences?
There are three reasons. Mathematical: squaring makes the loss function differentiable everywhere (absolute value has a kink at zero), which means gradient descent works smoothly. Statistical: if you assume the noise in your data is Gaussian (normally distributed), then MSE is the maximum likelihood estimator — it's the loss function that makes your data most probable under the model. Practical: squaring penalizes large errors more than small ones — being off by 10 costs 100, while being off by 1 costs 1. This means the model prioritizes fixing its biggest mistakes, which is usually what you want.
In a linear regression model predicting house price from square footage, the learned weight w₁ = 150 and bias b = 20000. What is the predicted price for a 1000 sq ft house?
y = w₁x₁ + b = 150 × 1000 + 20000 = 170,000. The weight is the price per square foot; the bias is the baseline price at zero square feet.
The Learning Process
Training linear regression with gradient descent is a loop: predict, measure error, adjust weights to reduce error, repeat. Each iteration, the gradient tells each weight how to change to reduce the loss. The process converges when the weights stop changing — meaning the loss is at a minimum.
For linear regression with MSE loss, the loss surface is convex — a single bowl shape. This means gradient descent is guaranteed to find the global minimum, regardless of where you start. This is a special property of linear regression that does NOT hold for neural networks.
1. Initialize weights (random or zeros)
2. For each training example:
- Predict: ŷ = X · w + b
- Calculate error: (y - ŷ)²
3. Update weights to reduce error (gradient descent)
4. Repeat until convergence
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.