The Problem
A PM ships a "research assistant" feature: users ask a question, the model answers with citations. In week two, a user posts a screenshot on social media — the assistant cited a peer-reviewed paper, complete with authors, journal name, and page numbers, that doesn't exist. It never did. The paper title sounds exactly like something that should exist, which is precisely why nobody caught it in review.
The support ticket lands on the PM's desk, and so does the obvious question from the VP: "Can we just switch to the bigger model?" The team burns a sprint swapping in the frontier model from the same vendor. The fabricated-citation rate drops a little. It does not go to zero, and a week later a different fabrication shows up — this time a plausible-sounding but nonexistent API method name in a coding-help feature built on the same upgraded model.
Both incidents have the same root cause, and it isn't model quality in the sense the VP means. An LLM generates text by predicting the most statistically likely next token given everything before it. It has no built-in step where it looks up "is this citation real" against a database and refuses to answer if the check fails. When the training distribution contains a lot of text that looks like "a citation belongs here," the model produces something citation-shaped — confidently, fluently, and with the same surface polish as a true statement. Scaling up the model makes the fabrication more fluent and less frequent in aggregate benchmarks. It does not add a fact-checking step that wasn't architecturally there before. This is the core thing a PM has to internalize: some failure modes are about model capability, and scale helps. Others are about model architecture, and scale doesn't touch them. Knowing which is which determines whether your fix is a model upgrade or a product change.
Hallucination and confabulation
"Hallucination" is the common term, but "confabulation" is more accurate: the model isn't perceiving something false, it's filling a gap with a plausible-sounding continuation, the same way a person with certain memory conditions confidently fabricates a detail rather than saying "I don't know." The model has no innate signal that distinguishes "I retrieved this fact from training data with high confidence" from "I'm pattern-matching the shape of a plausible answer." Both produce fluent, grammatically confident text. Unless a model is specifically trained or prompted to express uncertainty — and even then, imperfectly — it will produce an answer with the same tone whether it's right or fabricated.
This matters for product decisions: a feature that asks an LLM to state facts without grounding (no retrieval, no citation-checking, no verification step) will hallucinate at some nonzero rate no matter which vendor or model size you pick. The fix that actually works is architectural at the product level, not the model level: retrieval-augmented generation that grounds answers in a real, checkable source; a verification pass that checks generated claims against that source before showing them to a user; or an explicit abstention mechanism that rewards the model (via prompting or fine-tuning) for saying "I don't have a reliable source for that" instead of guessing.
Reasoning limits
LLMs can fail at multi-step logical or arithmetic reasoning, especially when a problem has distractor information, an unusual framing, or requires holding several intermediate facts in mind at once. A model can correctly solve a textbook version of a word problem and fail the same problem with one irrelevant sentence inserted, because it's pattern-matching against training-data structure, not executing a verified logical procedure the way a calculator or theorem prover does.
Chain-of-thought prompting and dedicated "reasoning" models (which generate extended intermediate reasoning before answering) measurably reduce this failure rate — they give the model more tokens to work through intermediate steps, which helps because each generated token becomes context for the next. But "reduce" is not "eliminate." Reasoning models still fail on sufficiently adversarial or long multi-step chains, and they can produce reasoning traces that sound rigorous while arriving at a wrong answer, which is arguably worse for a PM's postmortem than a plain wrong answer — the confident-looking work makes it harder for a user or reviewer to catch.
Knowledge cutoffs
Every model is trained on data up to some fixed date. It has no knowledge of anything after that date unless it's supplied through context — retrieval, browsing, or a user pasting it in. This sounds obvious, but the failure mode is subtle: models don't always reliably know or say when they're past their own cutoff. Ask about an event from after the cutoff and a model may hallucinate a plausible-sounding but fabricated answer rather than saying "I don't have information after my training cutoff," especially if the question is phrased as if the event is common knowledge.
This is a distinct failure mode from generic hallucination, and it matters because the fix is different and much simpler: give the model current information through retrieval or a tool call instead of relying on parametric memory. A "why didn't the AI know about X" support ticket about a recent event is a knowledge-cutoff gap, not a reasoning or scale problem, and no amount of "smarter model" closes a gap in what the model was ever shown.
Context rot
More context is not free. As a context window fills up — long documents, long conversation histories, many retrieved chunks — model attention and accuracy on the actually-relevant information tends to degrade, especially when the window contains irrelevant or repetitive content packed in alongside the useful parts. A model with a one-million-token context window doesn't use every one of those tokens with equal fidelity; performance on "find the needle in this haystack" tasks measurably drops as the haystack grows, and it drops faster when the haystack is noisy rather than clean.
The product implication: dumping an entire knowledge base into context "because the window is big enough now" is a worse strategy than retrieving a small number of precisely relevant chunks. Quality of what goes into context — tight retrieval, deduplication, summarization of stale turns — matters more than raw quantity. A PM scoping a RAG feature should treat "we have a huge context window" as permission to be lazier about retrieval precision only up to a point; past that point, more context measurably hurts the exact accuracy the feature depends on.
"Bigger model" is a fix for capability gaps, not architecture gaps
Some failures shrink with scale: general knowledge coverage, following complex instructions, coding correctness on average. Others don't, because they aren't caused by insufficient scale — they're caused by probabilistic generation with no built-in verification step. Hallucination without grounding, confident answers past a knowledge cutoff, and attention degradation over long noisy context all persist across model generations because the architecture hasn't changed, only its fluency has. Before you request a model upgrade, ask which category the bug is in. If it's architectural, the fix is RAG, verification, abstention prompting, or context curation — not a bigger bill.
A support ticket says a coding assistant "invented" a Python library method that doesn't exist, and it happened again after the team upgraded to the newest, largest model from the same vendor. What's the most likely explanation, and what's the correct product-level fix?
A fabricated API method name is confabulation, not a knowledge-cutoff or reasoning problem — the model is generating a plausible-sounding token sequence with no lookup step against real API surfaces, and this persists across model sizes because no LLM has a built-in "verify this exists" check. Rolling back doesn't address the root cause, and chain-of-thought helps with multi-step logic, not fact grounding. The actual fix is product-level: validate generated code against real documentation or a linter/compiler before it reaches the user, or use retrieval to ground code suggestions in the real API surface.
A user asks a deployed assistant about a product recall that was announced last week. The model confidently describes an "official recall process" that sounds plausible but is entirely fabricated — there's no evidence it retrieved anything about a real event. Is this a knowledge-cutoff issue or a hallucination issue, and what's the product fix?
The two failure modes compound here: the underlying gap is a knowledge cutoff (the event is after training data ends), but the visible symptom is hallucination, because the model fills the gap with a fabricated but fluent answer instead of expressing uncertainty. A newer training cutoff only pushes the same problem forward in time — it doesn't fix the pattern of confabulating instead of abstaining. The durable fix is retrieval/tool access for current information, combined with explicit abstention behavior (via prompting or fine-tuning) for whatever falls outside both the model's training data and what retrieval actually found.