Phase 1: AI & ML Literacy for PMs · 45 min · Anthropic Transparency Hub · Python
Model Capabilities & Limitations
A bigger model gets it wrong with more confidence, not less often enough to bet the roadmap on it.
Hiring signal: PMs who treat every LLM quality bug as 'the model needs to get smarter' escalate model-swap requests that don't fix anything and burn a quarter re-litigating the same incident. A PM who can diagnose whether a bad output is hallucination, a reasoning limit, a knowledge-cutoff gap, or context rot — and route each to the right product-level mitigation — is the one engineering trusts to scope AI features without reopening the same postmortem every release.
What you will learn
- Explain why LLMs hallucinate as a consequence of next-token prediction, not a fixable bug in one release
- Distinguish knowledge-cutoff gaps from hallucination and route each to the correct product fix
- Explain context rot and why adding more context to a prompt is not free
- Identify multi-step reasoning failure modes and why chain-of-thought reduces but doesn't eliminate them
- Explain why 'use a bigger model' fails to fix architectural failure modes and propose the correct product-level mitigation instead
The Problem
A PM ships a "research assistant" feature: users ask a question, the model answers with citations. In week two, a user posts a screenshot on social media — the assistant cited a peer-reviewed paper, complete with authors, journal name, and page numbers, that doesn't exist. It never did. The paper title sounds exactly like something that should exist, which is precisely why nobody caught it in review.
The support ticket lands on the PM's desk, and so does the obvious question from the VP: "Can we just switch to the bigger model?" The team burns a sprint swapping in the frontier model from the same vendor. The fabricated-citation rate drops a little. It does not go to zero, and a week later a different fabrication shows up — this time a plausible-sounding but nonexistent API method name in a coding-help feature built on the same upgraded model.
Both incidents have the same root cause, and it isn't model quality in the sense the VP means. An LLM generates text by predicting the most statistically likely next token given everything before it. It has no built-in step where it looks up "is this citation real" against a database and refuses to answer if the check fails. When the training distribution contains a lot of text that looks like "a citation belongs here," the model produces something citation-shaped — confidently, fluently, and with the same surface polish as a true statement. Scaling up the model makes the fabrication more fluent and less frequent in aggregate benchmarks. It does not add a fact-checking step that wasn't architecturally there before. This is the core thing a PM has to internalize: some failure modes are about model capability, and scale helps. Others are about model architecture, and scale doesn't touch them. Knowing which is which determines whether your fix is a model upgrade or a product change.
Hallucination and Confabulation
"Hallucination" is the common term, but "confabulation" is more accurate: the model isn't perceiving something false, it's filling a gap with a plausible-sounding continuation, the same way a person with certain memory conditions confidently fabricates a detail rather than saying "I don't know." The model has no innate signal that distinguishes "I retrieved this fact from training data with high confidence" from "I'm pattern-matching the shape of a plausible answer." Both produce fluent, grammatically confident text. Unless a model is specifically trained or prompted to express uncertainty — and even then, imperfectly — it will produce an answer with the same tone whether it's right or fabricated.
This matters for product decisions: a feature that asks an LLM to state facts without grounding (no retrieval, no citation-checking, no verification step) will hallucinate at some nonzero rate no matter which vendor or model size you pick. The fix that actually works is architectural at the product level, not the model level: retrieval-augmented generation that grounds answers in a real, checkable source; a verification pass that checks generated claims against that source before showing them to a user; or an explicit abstention mechanism that rewards the model (via prompting or fine-tuning) for saying "I don't have a reliable source for that" instead of guessing.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Reasoning Limits, Knowledge Cutoffs, Context Rot, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy