Objective
Learning objectives
- Define a series as the running sum of a sequence
- Explain why a moving average smooths a noisy sequence
- Implement a moving average function and apply it to a synthetic noisy sequence
- Compute a moving average by hand for the first few terms of a short sequence
Hook
Training loss almost never drops in a clean, obedient line — it jitters, step to step, even while genuinely trending downward. If you only ever look at the single most recent loss value, a normal noisy jump upward can look like the model got worse. Average your last 100 loss values together instead, and that same jump barely moves the line. Why does averaging specifically fix this, rather than just hiding real information?
See it
Here's 15 steps of a genuinely noisy — but genuinely decreasing — synthetic loss sequence, and its 3-step moving average alongside it:
raw = [1.869, 1.922, 1.735, 1.752, 1.663, 1.283, 1.157, 1.469, 1.08, 0.967, 1.248, 0.885, 0.968, 0.688, 0.67]
def moving_average(seq, window):
out = []
for i in range(len(seq)):
start = max(0, i - window + 1)
chunk = seq[start:i + 1]
out.append(round(sum(chunk) / len(chunk), 3))
return out
print(moving_average(raw, 3))
[1.869, 1.895, 1.842, 1.803, 1.717, 1.566, 1.368, 1.303, 1.235, 1.172, 1.098, 1.033, 1.034, 0.847, 0.775]
Look at step 6→7 in the raw sequence: 1.157 → 1.469, a sharp jump up, right in the middle of an overall downward trend — exactly the kind of single point that looks alarming in isolation. In the moving average, the same stretch reads 1.368 → 1.303, still calmly heading down. The jump didn't get deleted; it got outvoted by its two neighbors, which is precisely what averaging several nearby values does to any single noisy outlier.
Name it
A series is the running sum of a sequence — add up its terms as you go. A moving average (also called a running average) is a series' close cousin: instead of a running sum, it's a running average over a fixed-size window of the most recent terms, sliding forward one step at a time.
The window size is a real tradeoff, not a free parameter
A small window (like 3) stays responsive to genuine recent changes but leaves more noise visible. A large window (like 100) smooths aggressively but reacts slowly to a real, sudden shift — averaging in 99 old values means one new value barely moves anything. There's no universally correct window size; it depends on how noisy the data is and how quickly you need to notice a real change.
Code it
import matplotlib.pyplot as plt
steps = list(range(1, 16))
smoothed = moving_average(raw, 3)
plt.plot(steps, raw, marker="o", alpha=0.5, label="raw loss")
plt.plot(steps, smoothed, marker="o", label="3-step moving average")
plt.xlabel("training step")
plt.ylabel("loss")
plt.legend()
plt.savefig("loss_smoothing.png", dpi=150, bbox_inches="tight")
print(moving_average(raw, 5))
Widening the window from 3 to 5 makes the average even smoother — compare step 7's value, 1.518 here versus 1.368 with a window of 3, noticeably closer to its neighbors' overall trend and less pulled around by the single sharp 1.469 spike. Same function, same data, only the window size changed, exactly matching the tradeoff described in Name It.