Phase 0: Development Environment & Mathematical Foundations · 70 min · Python · NumPy · Matplotlib
The Concept
Why Probability Is the Language of ML
Machine learning is fundamentally about making predictions under uncertainty. When a classifier says "this email is 87% likely to be spam," that 87% is a probability. When a language model predicts the next word, it outputs a probability distribution over the entire vocabulary. When we train a model, we are finding parameters that maximize the probability of observing the training data. Every loss function, every evaluation metric, and every uncertainty estimate in ML is rooted in probability theory.
The three concepts you need before anything else: distributions (how probabilities are spread across outcomes), Bayes' theorem (how to update beliefs with evidence), and information theory (how to measure surprise and uncertainty). Let's build each one.
Probability Distributions: Modeling Uncertainty
A probability distribution assigns a probability to every possible outcome. Different distributions model different kinds of uncertainty:
- Bernoulli — a single yes/no outcome. Parameter $p$ = probability of "yes." Models: spam/not-spam, click/no-click, disease/no-disease
- Categorical — a single outcome from K possibilities. Models: which word comes next (vocabulary of 50,000+ words), which class an image belongs to
- Normal (Gaussian) — continuous values clustered around a mean. Parameters $\mu$ (mean) and $\sigma$ (spread). Models: measurement errors, feature distributions, weight initializations
- Beta — a distribution over probabilities themselves (values in [0,1]). Used as a prior in Bayesian inference: "I believe the click rate is around 3%, but I'm not certain"
The Normal distribution deserves special attention because it appears everywhere in ML. The famous bell curve: f(x) = (1 / (σ√2π)) * exp(-(x-μ)² / (2σ²)). When you initialize neural network weights, you sample from a normal distribution. When you assume your data has Gaussian noise, you're choosing MSE loss (because MSE is the maximum likelihood estimator for Gaussian noise). When you use batch normalization, you're forcing activations to be approximately normal.
import numpy as np
# Bernoulli: coin flip (0 or 1), parameter p = P(1)
p = 0.7
samples = np.random.binomial(1, p, size=10000)
print(f"Mean: {samples.mean():.3f} (should be ~{p})")
# Normal (Gaussian): bell curve, parameters μ (mean) and σ (std)
mu, sigma = 5.0, 2.0
samples = np.random.normal(mu, sigma, size=10000)
print(f"Mean: {samples.mean():.3f}, Std: {samples.std():.3f}")
# PDF: f(x) = (1 / (σ√2π)) * exp(-(x-μ)² / (2σ²))
def normal_pdf(x, mu, sigma):
return (1 / (sigma * np.sqrt(2 * np.pi))) * np.exp(-0.5 * ((x - mu) / sigma) ** 2)
You have a spam filter. P(spam) = 0.3, P(word "FREE" | spam) = 0.8, P(word "FREE" | not spam) = 0.1. What is P(spam | "FREE")?
By Bayes: P(spam|FREE) = P(FREE|spam) P(spam) / P(FREE) = 0.8 0.3 / (0.80.3 + 0.10.7) = 0.24 / 0.31 ≈ 0.77. The prior shifts the posterior away from the likelihood.
# Beta: distribution over probabilities (used as prior in Bayesian inference)
# Parameters α, β. Mean = α / (α + β)
alpha, beta = 3, 7 # prior belief that p ≈ 0.3
samples = np.random.beta(alpha, beta, size=10000)
print(f"Beta mean: {samples.mean():.3f} (should be ~{alpha/(alpha+beta):.3f})")
Bayes' Theorem: Updating Beliefs with Evidence
Bayes' theorem is the mathematical rule for updating your beliefs when new evidence arrives. You start with a prior belief, you observe evidence, and you update to a posterior belief. This is not just a formula — it is the fundamental model of how rational agents learn from data.
The intuition: if evidence is likely under your hypothesis AND unlikely otherwise, then seeing that evidence strongly supports your hypothesis. If the evidence is equally likely whether or not your hypothesis is true, it tells you nothing.
P(H|D) = P(D|H) * P(H) / P(D)
H = Hypothesis (e.g., "this email is spam")
D = Data (e.g., "email contains 'free money'")
P(H) = Prior: belief before seeing data
P(D|H) = Likelihood: how likely is this data if hypothesis is true?
P(H|D) = Posterior: updated belief after seeing data
P(D) = Evidence: normalizing constant (how likely is the data under all hypotheses?)
Every ML model is implicitly Bayesian. A classifier learns $P(\text{class} | \text{features})$ — the posterior probability of a class given the input. A language model learns $P(\text{next word} | \text{context})$. Training is the process of finding parameters that make the observed data most probable — which is Maximum Likelihood Estimation, the frequentist cousin of Bayes' theorem.
# Spam detection with Bayes
p_spam = 0.3 # prior: 30% of emails are spam
p_word_given_spam = 0.8 # P("free money" | spam)
p_word_given_ham = 0.01 # P("free money" | not spam)
# Posterior: P(spam | "free money")
p_word = p_word_given_spam * p_spam + p_word_given_ham * (1 - p_spam)
p_spam_given_word = (p_word_given_spam * p_spam) / p_word
print(f"P(spam | 'free money') = {p_spam_given_word:.4f}")
# → 0.9971 — almost certainly spam
Maximum Likelihood Estimation (MLE)
MLE finds the parameters that make the observed data most probable. This is what training a neural network actually does:
import numpy as np
# Suppose we observe 100 coin flips, 70 heads. What's the fair estimate of p?
# MLE: maximize P(data | p) = p^70 * (1-p)^30
# Take log: 70*ln(p) + 30*ln(1-p)
# Derivative: 70/p - 30/(1-p) = 0 → p = 70/100 = 0.7
# For a Gaussian: MLE of μ is the sample mean, MLE of σ² is the sample variance
data = np.random.normal(5.0, 2.0, size=1000)
mu_mle = data.mean()
sigma_mle = data.std()
print(f"MLE μ: {mu_mle:.3f}, MLE σ: {sigma_mle:.3f}")
# Connection to loss functions:
# - MSE loss = Gaussian negative log-likelihood (assuming constant variance)
# - Cross-entropy loss = Bernoulli/Categorical negative log-likelihood
# When you train a classifier with cross-entropy, you're doing MLE!
Information Theory: Measuring Surprise
Information theory gives us the vocabulary to talk about uncertainty and surprise quantitatively. The core idea: learning that a likely event happened tells you little; learning that an unlikely event happened tells you a lot. The amount of "information" in an event with probability $p$ is $-\log(p)$. A certain event ($p=1$) carries zero information. A rare event ($p=0.001$) carries about 10 bits of information.
Entropy is the average surprise of a distribution — how uncertain you are, on average, about what comes next. A fair coin has entropy 1 bit (maximum uncertainty for a binary outcome). A biased coin (99% heads) has entropy ~0.08 bits (you're almost always right, so there's little surprise). Entropy gives us a principled way to measure uncertainty, which is essential for deciding when a model is "confident" vs "guessing."
Cross-entropy measures the surprise you experience when the true distribution is $p$ but you predicted $q$. If your prediction matches reality ($p = q$), cross-entropy equals entropy — you're as surprised as you should be. If your prediction is wrong, cross-entropy exceeds entropy — you're more surprised than necessary. The gap between cross-entropy and entropy is the KL divergence: the "extra surprise" from using the wrong model. When you train a classifier with cross-entropy loss, you are literally minimizing the extra surprise between your model's predictions and the true labels.
import numpy as np
def entropy(p: np.ndarray) -> float:
"""Shannon entropy: average surprise of a distribution."""
p = p[p > 0] # remove zeros (0*log0 = 0)
return -np.sum(p * np.log2(p))
# Fair coin: maximum uncertainty
print(f"Entropy of fair coin: {entropy(np.array([0.5, 0.5])):.3f} bits") # 1.0
# Biased coin: less uncertainty
print(f"Entropy of biased coin: {entropy(np.array([0.99, 0.01])):.3f} bits") # 0.081
def cross_entropy(p: np.ndarray, q: np.ndarray) -> float:
"""Cross-entropy: surprise when true distribution is p but we model it as q."""
p = p[p > 0]
q = q[p > 0] # align with p's non-zero entries
return -np.sum(p * np.log2(q))
def kl_divergence(p: np.ndarray, q: np.ndarray) -> float:
"""KL divergence: how much information is lost when q approximates p."""
return cross_entropy(p, q) - entropy(p)
# Cross-entropy = entropy + KL divergence
# When p = q (perfect prediction), cross-entropy = entropy, KL = 0
p = np.array([0.7, 0.3])
q_perfect = np.array([0.7, 0.3])
q_bad = np.array([0.5, 0.5])
print(f"Perfect prediction CE: {cross_entropy(p, q_perfect):.4f}")
print(f"Bad prediction CE: {cross_entropy(p, q_bad):.4f}")
print(f"KL(perfect): {kl_divergence(p, q_perfect):.4f}") # 0
print(f"KL(bad): {kl_divergence(p, q_bad):.4f}")
Perplexity: Language Model Evaluation
# Perplexity = 2^cross_entropy — "how confused is the model?"
# Lower is better. A perplexity of 1 means perfect prediction.
# A perplexity of 10 means the model is as confused as if choosing uniformly from 10 options.
def perplexity(p: np.ndarray, q: np.ndarray) -> float:
return 2 ** cross_entropy(p, q)
# If true distribution is [0.7, 0.3] and model predicts [0.6, 0.4]:
print(f"Perplexity: {perplexity(np.array([0.7, 0.3]), np.array([0.6, 0.4])):.3f}")