Phase 0: Development Environment & Mathematical Foundations · 90 min · NumPy · Python · Matplotlib
The Problem
A junior engineer calls model.forward(x) without understanding what happens inside. When the model returns garbage, they can't tell whether the bug is in the embeddings, the attention weights, or the final projection — because to them it's all a black box. They don't know why their 768-dimensional vectors behave the way they do.
Linear algebra is the language neural networks speak. You don't need a math degree — you need four operations (dot product, norm, matrix multiply, softmax) and a feel for what they do. This lesson builds that feel one operation at a time, then assembles them into the attention mechanism that powers every transformer.
You already use this every day
When you search "comfortable running shoes" and get relevant results, a dot product ranked them. When ChatGPT decides which earlier words matter for the next token, it's a softmax over dot products. The math below isn't abstract — it's the engine under every AI product.
The Concept
The Four Objects of AI Math
Everything in AI is built from four kinds of objects. The only difference between them is how many indices you need to pick out a single number.
Scalar: 5 (a single number — 0 indices)
Vector: [3, 7, 2] (a list — 1 index: v[0]=3)
Matrix: [[1, 2], [3, 4]] (a grid — 2 indices: M[0][1]=2)
Tensor: [[[...], [...]], ...] (a stack of grids — 3+ indices)
A vector is just an ordered list of numbers, and in AI it represents meaning:
- A word embedding:
"king" → [0.2, -0.4, 0.8, ...] (often 768 numbers) - An image patch, a user's preferences, or a layer's weights
The magic is that similar things get similar vectors. Measuring "how similar" is the first operation we'll build.
Why Vectors Work: The Geometry of Meaning
When a neural network learns an embedding for the word "king," it assigns it 768 numbers. Those numbers define a point in a 768-dimensional space — a space you cannot visualize, but which has the same geometric rules as the 2D and 3D spaces you can. The key insight, proven empirically across thousands of models, is that training pushes semantically related words to nearby points in this space. "King" and "queen" end up close. "King" and "banana" end up far apart. This happens naturally as a byproduct of training on text — the model discovers that words used in similar contexts should have similar representations.
This geometric property is what makes vector search possible. To find documents relevant to a query, you embed both into the same space and measure how close they are. The "closeness" is measured with operations you'll build below: dot product, norm, and cosine similarity.
Matrix Multiplication as Geometric Transformation
The most important intuition in linear algebra for AI is this: a matrix is a function that transforms vectors. When you multiply a matrix $W$ by a vector $x$, you are applying a transformation — the matrix rotates, scales, and projects the vector into a new space.
In a neural network, each layer's weight matrix transforms the input vector into a new representation. The first layer might transform raw pixel values into edge detectors. The second transforms edges into shapes. The third transforms shapes into object parts. Each transformation is a matrix multiply followed by a nonlinearity (like ReLU). The network learns which transformations to apply by adjusting the matrix entries during training.
This is why matrix multiplication is not just "a bunch of multiplications and additions" — it is the mechanism by which a neural network progressively reshapes data into a form where the final answer (a classification, a prediction, a generated token) falls out naturally.
A 768-dimensional embedding for the word "king" is, concretely, what?
An embedding is a vector — an ordered list of numbers (here, 768 of them). Each number is one learned coordinate of the word's meaning.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.