Phase 2: Discovery & AI Product Strategy · 45 min · Anthropic Models Docs · OpenAI Models Docs · Python
The Problem
A PM has narrowed the policy-lookup feature to "Anthropic API + RAG" as the sourcing path, from the last lesson's TCO comparison. Now a more specific question lands: which model, from which vendor, at which tier? The team has three real candidates on the table — a frontier flagship model, a fast/cheap mid-tier model, and a competitor's flagship — and everyone in the room has a different opinion, none of them backed by the same evidence.
The engineer who ran a quick test in the vendor playground says the flagship "felt smarter." The finance partner has seen the per-token pricing and wants the cheapest option, full stop. The PM, without a documented framework, is stuck mediating a debate between "it felt smarter" and "it's cheapest" — two answers to two different questions, neither of which is "which model actually fits this feature's requirements."
This is a scoping failure the previous two lessons have already built the muscle to avoid: just as a build-vs-buy decision needs a real TCO comparison instead of a vendor preference, a model-selection decision needs a real weighted comparison across the axes that actually matter for the specific feature — not a single impression from a demo, and not cost in isolation. This lesson builds that comparison.
The four core axes
Nearly every model-selection decision comes down to trading off four things, and a feature's specific requirements determine how much each one should be weighted:
- Cost — per-token pricing for input and output, which compounds directly with the volume projections from the TCO lesson. A model that's twice as expensive per token isn't automatically wrong, if it earns that cost back in quality or reduced error-handling burden — but the tradeoff has to be made explicitly, not ignored.
- Latency — how fast the model responds, which matters enormously differently depending on the feature. A batch job that runs overnight can tolerate a slow, high-quality model. A live chat interface where a user is waiting on screen cannot — every extra second of latency there is a measurable UX cost, sometimes measured in abandonment rate.
- Quality — how well the model actually performs on your specific task, which (per the model-card lesson) is not the same as leaderboard rank. This has to be measured on your own eval, even a rough one, not assumed from a benchmark score or a demo impression.
- Context window — whether the model can hold everything a single call needs (per the LLM fundamentals lesson), which is a hard feasibility gate, not just a quality dial. A model that's cheaper and faster but can't fit your typical input isn't a real candidate at all for that feature.
These four axes pull against each other in practice: the fastest, cheapest models are usually smaller and lower-quality on complex tasks; the highest-quality frontier models usually cost more and respond slower; long-context models often carry a cost or latency premium over their shorter-context siblings from the same vendor. There is no model that wins on all four simultaneously — which is exactly why the weighting has to be feature-specific.
The same four candidates can have different "winners" for different features
A live customer-facing chat feature might weight latency and cost heavily, with quality needing to only clear a "good enough" bar — favoring a fast mid-tier model. A once-a-day compliance-document review feature might weight quality heavily and barely care about latency at all — favoring the frontier flagship, even at higher per-token cost, because the volume is low and the cost of a quality miss is high. Running the same four candidate models through two different weightings for two different features can produce two different "best" answers, and that's not inconsistency — that's the matrix working correctly. A single company-wide "which model do we use" policy without per-feature weighting is a red flag, not a simplification.
Building a weighted scoring matrix
A defensible model-selection matrix needs three things: real candidate models with real (or realistically estimated) numbers on each axis, a feature-specific weighting that reflects what actually matters for this use case, and normalized scores so a model that dominates in one raw number (huge context window) doesn't automatically dominate the weighted total if that axis isn't actually a priority for this feature.
The mechanics: score each model 0-100 on each axis (cost and latency scored so lower raw cost/latency produces a higher score — cheaper and faster is better; quality and context window scored directly), multiply each axis score by its feature-specific weight (weights summing to 1.0 across the four axes), and sum to a weighted total. The model with the highest weighted total is the recommendation — but the real deliverable isn't the number, it's the matrix itself, because that's what lets an engineering lead check the reasoning, disagree with a specific weight, and adjust it, instead of just accepting or rejecting a conclusion they can't audit.
A team is choosing a model for a live customer support chat widget. Users are actively waiting on screen for a response, cost needs to stay low because volume is very high (millions of messages/month), and the task itself (answering common questions from a knowledge base) is not especially difficult reasoning. Candidate A is a frontier flagship: highest quality, highest cost, highest latency. Candidate B is a fast/cheap mid-tier model: good-enough quality on this specific task, lowest cost, lowest latency. How should the weighting be set, and which model should win?
The feature's actual requirements -- live user-facing latency sensitivity, very high volume driving cost sensitivity, and a task that doesn't require frontier-level reasoning -- point directly to weighting cost and latency heavily while treating quality as a "good enough" gate rather than the dominant axis. Candidate B fits this weighting even though it scores lower on raw quality, because that extra quality isn't needed for answering common knowledge-base questions and isn't worth the added cost and latency at this volume. Weighting quality highest "always" or weighting all axes equally ignores that the right weighting is feature-specific, which is the core lesson here -- and context window isn't the binding constraint in this scenario since nothing suggests either candidate fails to fit the required input.
Documenting the choice so engineering can review the reasoning
A model selection that shows up as "we chose Model B" in a planning doc, with no visible matrix, forces an engineering lead to either trust the PM blindly or redo the analysis themselves — neither is a good outcome. The matrix itself, including the raw numbers, the weights, and the reasoning behind the weights, is the actual deliverable. This does three things a bare conclusion can't: it lets an engineering lead spot-check a number they know better than the PM does (actual observed latency, real pricing), it lets someone push back on a specific weight ("I think context window should matter more here because of X") without re-litigating the whole decision, and it creates a paper trail for revisiting the decision later — because model selection is not a permanent decision. A new model release, a usage pattern that shifts (the feature starts handling longer documents than expected), or a cost structure change from a vendor are all legitimate triggers to re-run the same matrix with updated numbers, not to start the debate over from scratch.
Six months after a model selection matrix recommended Candidate B (fast/cheap mid-tier) for the support chat widget, the product has added a new capability: summarizing entire multi-day support threads (up to 40,000 tokens) before responding, which Candidate B's context window can't fit. An engineer says "we already decided on Candidate B, let's not reopen this." What's the right response?
The lesson is explicit that model selection is not a permanent decision -- a new model release, a shifting usage pattern, or a vendor cost change are all legitimate triggers to re-run the matrix. Here, the usage pattern has materially changed: the feature now requires fitting up to 40,000 tokens, which may exceed Candidate B's context window entirely, turning what was a quality/cost/latency tradeoff into a hard feasibility failure. Refusing to reopen the decision because "we already decided" ignores that the original decision was scoped to different requirements than the ones the feature now has.