Phase 2: Discovery & AI Product Strategy · 45 min · Anthropic Models Docs · OpenAI Models Docs · Python

Model Selection Tradeoffs

The best model on the leaderboard is rarely the best model for your latency budget, your cost ceiling, and your context window need, all at once.

Hiring signal: AI PM total comp research shows the technical bar is API/integration literacy and defensible tradeoff judgment, not benchmark trivia — an engineering lead expects a PM to walk into a model-selection conversation with a documented, weighted comparison across cost, latency, quality, and context window, not a preference based on which model 'felt smarter' in a demo.

What you will learn

Identify cost, latency, quality, and context window as the four core axes of a model-selection decision
Build a weighted scoring matrix across candidate models that reflects a specific feature's priorities, not a generic ranking
Explain why the same four candidate models can produce different 'winners' for two different features
Defend a model selection to an engineering lead using the matrix's weighting and scores, not benchmark leaderboard rank alone
Identify when a model selection decision should be revisited (a new model release, a usage pattern shift) rather than treated as permanent

The Problem

A PM has narrowed the policy-lookup feature to "Anthropic API + RAG" as the sourcing path, from the last lesson's TCO comparison. Now a more specific question lands: which model, from which vendor, at which tier? The team has three real candidates on the table — a frontier flagship model, a fast/cheap mid-tier model, and a competitor's flagship — and everyone in the room has a different opinion, none of them backed by the same evidence.

The engineer who ran a quick test in the vendor playground says the flagship "felt smarter." The finance partner has seen the per-token pricing and wants the cheapest option, full stop. The PM, without a documented framework, is stuck mediating a debate between "it felt smarter" and "it's cheapest" — two answers to two different questions, neither of which is "which model actually fits this feature's requirements."

This is a scoping failure the previous two lessons have already built the muscle to avoid: just as a build-vs-buy decision needs a real TCO comparison instead of a vendor preference, a model-selection decision needs a real weighted comparison across the axes that actually matter for the specific feature — not a single impression from a demo, and not cost in isolation. This lesson builds that comparison.

The Four Core Axes

Nearly every model-selection decision comes down to trading off four things, and a feature's specific requirements determine how much each one should be weighted:

Cost — per-token pricing for input and output, which compounds directly with the volume projections from the TCO lesson. A model that's twice as expensive per token isn't automatically wrong, if it earns that cost back in quality or reduced error-handling burden — but the tradeoff has to be made explicitly, not ignored.
Latency — how fast the model responds, which matters enormously differently depending on the feature. A batch job that runs overnight can tolerate a slow, high-quality model. A live chat interface where a user is waiting on screen cannot — every extra second of latency there is a measurable UX cost, sometimes measured in abandonment rate.
Quality — how well the model actually performs on your specific task, which (per the model-card lesson) is not the same as leaderboard rank. This has to be measured on your own eval, even a rough one, not assumed from a benchmark score or a demo impression.
Context window — whether the model can hold everything a single call needs (per the LLM fundamentals lesson), which is a hard feasibility gate, not just a quality dial. A model that's cheaper and faster but can't fit your typical input isn't a real candidate at all for that feature.

These four axes pull against each other in practice: the fastest, cheapest models are usually smaller and lower-quality on complex tasks; the highest-quality frontier models usually cost more and respond slower; long-context models often carry a cost or latency premium over their shorter-context siblings from the same vendor. There is no model that wins on all four simultaneously — which is exactly why the weighting has to be feature-specific.

The same four candidates can have different "winners" for different features
A live customer-facing chat feature might weight latency and cost heavily, with quality needing to only clear a "good enough" bar — favoring a fast mid-tier model. A once-a-day compliance-document review feature might weight quality heavily and barely care about latency at all — favoring the frontier flagship, even at higher per-token cost, because the volume is low and the cost of a quality miss is high. Running the same four candidate models through two different weightings for two different features can produce two different "best" answers, and that's not inconsistency — that's the matrix working correctly. A single company-wide "which model do we use" policy without per-feature weighting is a red flag, not a simplification.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Building a Weighted Scoring Matrix, Documenting the Choice So Engineering Can Review the Reasoning, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.