Phase 2: Discovery & AI Product Strategy · 45 min · OpenAI API · Anthropic API · Google Gemini API
The Problem
The policy-document lookup opportunity from the last lesson gets approved. Now someone has to decide how to actually source the AI capability. The engineering lead wants to fine-tune an open-weight model on the company's policy corpus, arguing it'll be cheaper at scale and give more control. The CTO wants to just call the Anthropic API with a well-built RAG pipeline, arguing speed to market matters more than marginal cost right now. A vendor cold-emails the VP of Product pitching a fully managed "AI knowledge base" SaaS product that does the whole thing for a flat monthly fee.
All three are legitimate options. None of them is obviously right without a cost model. This is where a lot of AI product work goes wrong: the decision gets made on whoever argued most persuasively in the room, or on which option the loudest engineer personally wants to work on, instead of on a total cost of ownership comparison that accounts for what actually changes at 10x usage.
The build-vs-buy-vs-fine-tune decision is really three separate axes collapsed into one conversation, and untangling them is the first job:
- Where does the capability come from? A frontier API (OpenAI, Anthropic, Google), an open-weight model you host yourself (Llama, Mistral, DeepSeek, Qwen), or a fully managed third-party SaaS product.
- How do you adapt it to your problem? Prompting alone, retrieval-augmented generation (grounding responses in your own data without touching model weights), or fine-tuning (actually updating weights on your own examples).
- Who owns the operational burden? Do you run inference infrastructure, or does a vendor?
The vendor landscape
The frontier API providers — OpenAI, Anthropic, Google — sell access to state-of-the-art models per token, with no infrastructure to manage. You get frequent model upgrades, strong general reasoning, and predictable per-token pricing you can find on their public pricing pages. The tradeoff is per-token cost at scale, less control over the exact model version and its update cadence, and data-handling terms you have to read carefully if you're in a regulated industry.
Open-weight models — Llama, Mistral, Qwen, DeepSeek, and others — flip that tradeoff. You self-host (on your own GPUs or a cloud GPU rental), which means no per-token API fee, full control over the model version (it never silently changes under you), and the ability to fine-tune freely. The cost shifts from "pay per token" to "pay for GPU capacity whether you're using it or not," plus the engineering time to actually run inference infrastructure reliably. This is a real fixed-cost-vs-variable-cost tradeoff, not just a technical one — it only pays off once usage is high and predictable enough that the fixed GPU cost beats the variable per-token cost.
A fully managed vendor SaaS product sits on top of either of these — it's "buy" in the purest sense. You give up the most control and the most differentiation (a competitor can buy the exact same product), but you get the fastest time to a working feature and someone else owns the reliability and security burden.
Fine-tuning is a sourcing decision, not a capability upgrade
Product teams often treat fine-tuning as "make the model smarter," but the mechanism is narrower and more specific: fine-tuning changes how a model responds to your specific input distribution and desired output format, using examples you provide. It rarely teaches a model new knowledge it didn't already have — that's what retrieval (RAG) is for. If the goal is "answer questions about our specific, changing policy documents," RAG is almost always the right tool, because fine-tuning bakes a knowledge snapshot into the weights that goes stale the moment a policy changes, and re-fine-tuning on every update is far more expensive than re-indexing a document store. Fine-tuning earns its cost when the goal is a consistent format, tone, or behavior pattern across thousands of examples — not when the goal is "know about our current data."
Total cost of ownership: what actually changes at scale
A TCO comparison that only looks at per-token API pricing is incomplete, and it's the single most common mistake in this decision. The real comparison has to include:
- Inference cost — per-token API pricing (frontier) or GPU-hour cost amortized per request (self-hosted)
- Fine-tuning cost — a one-time or periodic training cost, which exists for both frontier providers' fine-tuning APIs and self-hosted setups, but scales very differently (self-hosted fine-tuning needs GPU time and ML engineering hours; API fine-tuning is billed per training token but needs no infrastructure)
- Hosting/ops cost — near-zero for a frontier API, real and ongoing for self-hosted (GPU reservations, scaling, monitoring, on-call)
- Engineering time — the cost most PMs underweight. A frontier API integration might take an engineer two weeks. A self-hosted fine-tuned model might take an ML engineer two months to get to production-grade reliability, and that's fully loaded salary cost, not a rounding error.
The reason this matters for a PM specifically: at low volume, a frontier API almost always wins on total cost, because the fixed engineering and infrastructure cost of self-hosting doesn't amortize over enough requests to beat a small per-token bill. At high, predictable volume, the crossover can flip — this is exactly the kind of threshold an engineering lead expects a PM to be able to reason about, even approximately, before agreeing to a roadmap commitment.
A PM is comparing sourcing options for a feature projected at 500,000 requests/month at launch, growing to 8 million requests/month within a year. The frontier API costs scale linearly with volume. The open-weight self-hosted option has a large fixed monthly GPU cost that doesn't change much with volume, plus a one-time $40k engineering cost to stand it up. At launch volume, the frontier API is cheaper. What is the single most important thing the PM needs to model before recommending the frontier API long-term?
This is exactly the crossover analysis a TCO comparison exists to catch. A per-token cost that's clearly cheaper at 500k requests/month can become dramatically more expensive than a fixed-cost self-hosted setup at 8 million requests/month, because the API cost scales linearly with volume while the self-hosted cost is largely fixed. Recommending the frontier API without modeling this crossover point risks locking the company into a sourcing decision that gets expensive right as the feature succeeds -- the PM needs to project cost at launch, growth, and scale volumes (the same three-stage projection used in AI PRDs) and show where, if anywhere, the lines cross.
A team is scoping an internal tool that drafts first-pass responses to routine vendor-contract questions. Legal wants consistent phrasing and a fixed internal tone across every response, and the underlying contract terms rarely change once a template is finalized. An engineer proposes fine-tuning an open-weight model on 3,000 examples of legal-approved responses. A PM pushes back and asks whether RAG would work instead. Whose framing is right, and why?
The lesson's callout draws exactly this distinction: fine-tuning earns its cost when the goal is a consistent format, tone, or behavior pattern across many examples — not when the goal is "know about our current data." Here, the goal is explicitly consistent phrasing and tone, and the underlying knowledge (contract terms in a finalized template) is stable, not constantly changing, so the usual fine-tuning risk (the trained-in knowledge going stale) is low. This is one of the narrower cases where fine-tuning is a defensible choice over RAG, unlike the policy-lookup example earlier in the lesson where the underlying knowledge changes often.
Making the recommendation
The output of this decision isn't "we chose X" — it's a documented comparison an engineering lead can sanity-check and a finance stakeholder can trust. That means the same volume assumptions applied consistently across all three options, the fine-tuning question answered explicitly (do we need it, or does RAG cover the requirement), and a stated crossover point if one exists. This lesson's artifact builds exactly that comparison, and it feeds directly into Lesson 3's model-selection matrix once a sourcing path is chosen.