The Problem
A PM is scoping a feature: upload a policy document, ask the AI questions about it, get accurate answers with citations. The policy document is 200 pages. In the spec, the PM writes one line: "the AI reads the document and answers questions about it." It sounds obviously feasible — humans read 200-page documents all the time, and the demo with a 10-page sample PDF worked perfectly.
It ships. Within a week, two things go wrong. First, real customer documents are 3-4x longer than the demo file, and some uploads fail outright with an opaque API error — the document plus the conversation history no longer fits in a single call to the model. Second, finance flags the AI feature's bill: it's costing 15x what was estimated, because every single question sent the entire document back to the model as input, every time, and nobody priced that out per token before launch.
Both problems trace back to the same gap: the PM's spec assumed "the AI can just read the whole document" the way a person does — sequentially, once, holding it in memory for free. That's not how it works. An LLM call has a hard ceiling on how much text it can process at once (the context window), and every token of input and output has a real, metered cost. "The AI reads the document" is not a feasible-or-not question you can answer by demoing on a short file. It's a token math question, and it has to be answered before the spec ships, not after the bill arrives.
This lesson is that math — not the deep architecture, but the four concepts (tokens, context windows, embeddings, attention) you need to catch this class of mistake before it becomes a production incident.
Tokens: the actual unit of everything
LLMs don't process text as words or characters. They break text into tokens — sub-word chunks, roughly whole common words, word fragments, punctuation, or parts of longer/rarer words. "Tokenization" is common, but might become "token" + "ization" or something similar depending on the model's vocabulary. This isn't a minor implementation detail: token count, not word count and not character count, is what every LLM API meters. Context window limits are stated in tokens. Pricing is stated in dollars per million tokens. If you estimate feasibility or cost using word counts, you will be wrong, sometimes by a lot, because token-per-word ratios vary with language, formatting, and content type (code and non-English text often tokenize less efficiently than plain English prose).
The rule of thumb for English prose: roughly 4 characters per token, or roughly 0.75 tokens per word. That's a planning heuristic, not a guarantee — dense technical text, tables, and unusual formatting can skew it in either direction. For anything you're actually going to spec or price, don't eyeball it.
Go count real tokens before you scope anything
Open https://platform.openai.com/tokenizer right now and paste in a real document you'd expect a feature to process — an email, a support ticket, a page from a policy doc. Look at two numbers: the token count, and how the tool visually chunks the text into pieces. You'll see tokens don't line up with word boundaries the way you'd assume, and you'll get a real per-document token count instead of a guess. Do this before you write "the AI reads X" in any spec — it takes thirty seconds and it's the single highest-leverage habit in this lesson.
Context windows: the ceiling you can't spec around
The context window is the maximum number of tokens a model can attend to in a single call — and depending on the model, that ceiling covers input and output combined. A model with a 128K-token context window doesn't give you 128K tokens of document plus unlimited room to answer; the system prompt, the conversation history, the retrieved document chunks, and the model's own response all draw from the same budget. Run the numbers and the "just read the whole document" plan often doesn't fit: a 200-page policy document at roughly 500 tokens per page is ~100,000 tokens before you've added a system prompt, a user question, or left any room for a multi-turn conversation.
This is why context window size is a real product constraint, not a spec-sheet footnote you skim past. It determines whether a feature idea is buildable as "just paste the whole thing in" or whether it requires chunking — splitting a long document into pieces, processing or retrieving only the relevant pieces per request (this is the core idea behind RAG, covered in a later lesson) — or summarization passes, where the document is condensed before the model reasons over it. Chunking and retrieval add real engineering complexity: more moving parts, more places for accuracy to degrade, more to test. A PM who checks document-size-in-tokens against context-window-size-in-tokens during scoping can flag "this needs chunking" as a design requirement on day one, instead of discovering it as a production bug after a customer's real document fails to upload.
It's also worth knowing that a model can technically accept a huge context window and still perform worse on information buried in the middle of a very long input than on information near the start or end — sometimes called "context rot," covered in the next lesson. Fitting in the context window is necessary but not sufficient for a feature to actually work well.
A PM is scoping a feature that summarizes customer support call transcripts. A typical call transcript is about 12,000 tokens. The model being used has a 32,000-token context window (input + output combined). The feature also needs to include the last 3 transcripts as context for tone/history, plus a system prompt (~500 tokens) and a summary output (~800 tokens). Will a single call with all 3 transcripts plus the new one fit?
4 transcripts at ~12,000 tokens each is already ~48,000 tokens — over the 32,000-token window before the system prompt (~500) or the output (~800) are even added. This is exactly the kind of thing that looks fine in a spec ("include recent history for context") and breaks in production. The fix is a design decision, not a bug fix: summarize each prior transcript down to a short digest before including it, retrieve only the most relevant prior transcript instead of all 3, or use a model with a larger context window and re-run the cost math (larger windows are usually priced accordingly).
Embeddings: finding meaning, not just matching words
An embedding is a numerical vector representation of a piece of text (or an image, or audio) that captures its meaning, positioned in a high-dimensional space such that texts with similar meaning end up close together, even if they don't share any of the same words. "The customer wants a refund" and "requesting my money back" use almost no overlapping vocabulary, but their embeddings land near each other, because they mean roughly the same thing.
That property is what makes embeddings useful for search and retrieval. A traditional keyword search for "refund" would miss "requesting my money back" entirely. An embedding-based (semantic) search finds it, because it's comparing meaning, not matching strings. This is the mechanism underneath most "search your documents with AI" and "chat with your data" features, and it's the retrieval half of RAG (retrieval-augmented generation): instead of stuffing an entire knowledge base into a context window, you embed the user's question, find the most semantically similar chunks of your documents, and pass only those chunks to the model. Embeddings are also how many recommendation and deduplication features work — "find things similar to this one" is fundamentally an embedding-distance calculation.
At the PM level, the thing worth internalizing is: embeddings let a feature find similar-meaning content even when the exact words differ, and that's the building block that makes "search my documents," "find related tickets," and "retrieve just the relevant part of this huge knowledge base" possible without ever fitting the whole knowledge base into a single context window.
A product team wants to build "find similar past support tickets" so agents can see how similar issues were resolved before. A junior engineer proposes matching tickets by exact keyword overlap in the ticket title. Why would an embeddings-based approach likely work better for this feature?
Keyword matching only catches tickets that share literal words — it would miss the connection between "app crashes on login" and "cannot sign in, freezes immediately" even though a human agent would immediately recognize them as the same underlying issue. Embeddings represent meaning as vectors, so semantically similar tickets land close together in that vector space regardless of exact wording. This is precisely the gap embeddings close, and it's why "similar ticket" or "similar document" features are almost always built on embeddings rather than keyword matching once you need to handle real-world phrasing variation.
Transformer architecture: attention, at PM depth
You don't need the matrix math to reason about LLM behavior as a PM, but it helps to understand the one core idea that makes modern LLMs work: attention. When a transformer model is predicting the next token, it doesn't process the input strictly left to right and forget everything behind it, the way a simple word-by-word model would. Instead, for every token it's about to generate, it looks back across all the other tokens in its context window and computes how much each one should matter — how much "attention" to pay to each — for producing the next token correctly.
This is the mechanism that lets a model reference something you said early in a long conversation, connect a pronoun ("it") back to the right noun several paragraphs earlier, or notice that a constraint mentioned on page 3 of a document is relevant to a question about page 40. It's also, mechanically, why context window size matters so much: attention operates over whatever's in the context window, so anything outside that window simply isn't available to attend to — it doesn't exist to the model for that call, no matter how relevant it might be.
For a PM, the practical takeaway is: attention is what makes long-range coherence possible, and it's also expensive to compute — attention cost grows faster than linearly with the number of tokens in the context, which is part of why very large context windows are harder and more expensive to serve, and why performance can still degrade on information buried deep in a long input even when it technically "fits." You don't need to compute this yourself. You do need to know that "give the model more context" isn't free, isn't infinite, and doesn't guarantee the model weighs everything in that context equally well.