The Concept
Decision Trees: Learning by Asking Questions
A decision tree splits data into regions by asking yes/no questions. At each node, it chooses the question that makes the resulting groups as "pure" as possible — meaning each group contains mostly one class. "Is income > 50k?" splits the data into two groups; if high-income customers mostly churn and low-income ones mostly don't, that's a good split.
[income > 50k?]
/ \
yes no
[age > 30?] predict: churn
/ \
no-churn churn
The tree keeps splitting until every leaf is pure (all one class) or a stopping criterion is reached (max depth, min samples per leaf). The measure of "purity" is either Gini impurity or entropy for classification, and variance for regression. A split is chosen to maximize the reduction in impurity — the "information gain."
Why Single Trees Overfit
A single decision tree will keep splitting until every leaf is pure. On a training set of 1000 samples, a deep enough tree can create a unique leaf for every single sample — 100% training accuracy. But this is memorization, not learning. The tree has learned the noise in the training data, not the underlying pattern. On new data, it performs poorly. This is the classic low bias, high variance problem: the model is flexible enough to fit anything (low bias), but its predictions change dramatically depending on which training examples it saw (high variance).
The Two Great Ideas: Bagging and Boosting
The two ensemble methods fix this overfitting problem in fundamentally different ways:
Bagging (Random Forest) reduces variance by averaging. Train many independent trees on different random subsets of the data (bootstrap sampling) and random subsets of features, then average their predictions (or take a majority vote). Because each tree sees different data and different features, they make different errors — and averaging cancels out the errors. The key insight: averaging reduces variance without increasing bias. A forest of 100 deep trees, each overfitting on different data, produces a stable, generalizable prediction.
Boosting (XGBoost, LightGBM) reduces bias by correcting. Train trees sequentially: the first tree makes predictions, the second tree is trained on the residuals (errors) of the first, the third on the residuals of the second, and so on. Each tree corrects the mistakes of the ensemble so far. The final prediction is a weighted sum of all trees. Boosting works because even if each individual tree is simple (a "weak learner" — slightly better than random), the sequence of corrections builds a strong model. This is why boosting typically outperforms bagging on tabular data: it directly attacks the bias problem.
Bagging (Random Forest): train many trees on bootstrapped samples + random feature
subsets, then AVERAGE. -> reduces VARIANCE. Trees are independent, trained in parallel.
Boosting (XGBoost/LGBM): train trees SEQUENTIALLY, each correcting the previous one's
errors. -> reduces BIAS (and variance). Usually the strongest tabular model.
Your tabular dataset has high bias (underfitting) — even a deep tree can't capture the pattern. Which ensemble approach is more likely to help: Random Forest or XGBoost?
Random Forest (bagging) reduces variance by averaging independent trees — it won't help if the base model underfits. XGBoost (boosting) trains trees sequentially, each correcting the previous one's errors, directly reducing bias. For underfitting, boosting is the right tool.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.