The Problem
A product team wants to build "ask questions about our internal engineering wiki." In the kickoff meeting, three people propose three different builds. The ML engineer wants to fine-tune an open-weight model on the wiki's full text, arguing "then the model will just know it." The founding engineer wants to build a multi-step agent that can search the wiki, cross-reference Jira tickets, and post a summary to Slack, because "agents are what everyone's building now." The PM, unsure which is right, agrees to a timeline that tries to accommodate a version of all three.
Eight weeks later: the fine-tune is stale because the wiki gets edited weekly and nobody wants to re-run a training job every week. The agent's multi-step Jira-cross-referencing capability is barely used — most questions are single-lookup, not multi-step tasks. The team ships a much smaller RAG pipeline (retrieve relevant wiki pages, pass them to the model, answer the question) in the final two weeks, and it's the part of the build that actually works.
This is not a story about bad engineers. It's a story about skipping the one question that should have been asked in the kickoff meeting: what, specifically, is the gap between what the model can already do and what this feature needs? RAG, fine-tuning, and agents close three genuinely different gaps. Reaching for the wrong one doesn't just waste budget — it ships a feature that's more complex than the problem required, and complexity is where AI features go to fail quietly.
What each approach actually changes
RAG (retrieval-augmented generation) doesn't change the model at all. It changes what the model sees at the moment it answers: instead of relying on what's baked into the model's weights from training, RAG retrieves relevant chunks of your own data (documents, tickets, product data) and inserts them into the prompt as context. The model then answers grounded in that retrieved content. RAG is the right tool when the gap is knowledge — the model doesn't know something specific to you, and that something changes often enough that baking it into weights would go stale.
Fine-tuning changes the model's weights using examples you provide. It doesn't reliably teach new facts (as covered in the previous lesson on build vs buy) — it changes how the model responds: the format, tone, structure, or behavior pattern it defaults to for a given kind of input. Fine-tuning is the right tool when the gap is consistency of behavior — you need the model to reliably produce a specific output shape or voice across thousands of similar cases, and prompting alone hasn't gotten it reliable enough.
Agents don't change what the model knows or how it's tuned — they change what the model is allowed to do. An agentic system gives the model access to tools (search, code execution, API calls) and lets it decide, across multiple steps, which tools to call and in what order to accomplish a task. Agents are the right tool when the gap is multi-step task execution — the job genuinely requires deciding a sequence of actions based on intermediate results, not just answering a single question.
These three are not mutually exclusive. A production system often combines them: an agent that uses RAG as one of its tools, running on a model that's been fine-tuned to format its final answers a specific way. The mistake in the opening scenario wasn't using any one of these — it was reaching for all three before establishing which gap the feature actually had.
Start from the gap, not the technique
Before proposing RAG, fine-tuning, or an agent, answer one question: is the problem that the model doesn't know something (RAG), that its behavior/format isn't consistent enough (fine-tuning), or that the task requires multiple decided steps rather than one answer (agent)? If you can't name which gap you're closing, you're choosing a technique because it's trendy, not because it fits — and that's exactly how a team ends up building three things when the problem needed one.
Cost and complexity, roughly ordered
The three approaches also differ sharply in what they cost to build and maintain, and the ordering matters for a roadmap conversation:
- Prompting alone (no RAG, no fine-tune, no agent) is the cheapest and fastest to test — often hours to days. Always the first thing to try, and often sufficient for surprisingly complex-sounding tasks.
- RAG adds a retrieval pipeline: a vector database or search index, an embedding step, and retrieval logic. Meaningfully more engineering than prompting alone (typically weeks, as seen in the build-vs-buy TCO lesson), but it stays current automatically as the underlying documents are updated — no retraining required.
- Fine-tuning adds a training pipeline: curated examples, a training run, and evaluation to confirm the fine-tune actually improved the target behavior without degrading general capability. This is usually the most expensive of the three per unit of behavior change, and it needs to be re-run whenever you want to adjust behavior — there's no "just edit a document" equivalent.
- Agents add orchestration complexity: tool definitions, error handling when a tool call fails or returns something unexpected, guardrails to prevent an agent from taking an unintended action, and often a human-approval step for consequential actions. Agents also compound the failure modes from the previous lesson — a multi-step agent that hallucinates at step 2 can carry that error into steps 3 through 7 before anyone notices.
The practical ordering for a PM defending a build plan: try prompting first, add RAG if the gap is current/specific knowledge, add fine-tuning only if prompting plus RAG still can't hit a consistent output format or tone after real iteration, and reach for a full agent only when the task genuinely can't be done as a single well-grounded prompt — not because "agent" sounds more sophisticated in a roadmap slide.
A customer support team wants an AI feature that reads an incoming ticket, looks up the customer's account status and recent order history from two different internal systems, decides whether the issue qualifies for an automatic refund under company policy, and if so, issues the refund and replies to the customer — all without human intervention for straightforward cases. Which approach is the right foundation, and why?
This scenario has all the hallmarks of an agentic task: it requires querying two different systems, applying conditional logic, and taking a real-world action (issuing a refund), not just producing an answer to a question. Fine-tuning wouldn't give the model live access to account systems, and RAG alone only retrieves and answers -- it doesn't take multi-step actions or call external systems. The correct design uses an agent architecture where RAG-style retrieval (pulling the refund policy) is one tool among several the agent uses, but the task's defining characteristic -- deciding what to do next based on what it just found out -- is what makes this an agent problem, not a RAG or fine-tuning problem.
When each approach is over-engineering
The reverse mistake — over-building — is just as costly as picking the wrong approach, and it's more common than it should be given how much attention agents get right now.
Reaching for an agent when a single well-grounded prompt would work adds orchestration complexity, more failure surface, and slower responses for no product benefit — if the task is "answer this one question using this one retrieved document," an agent's multi-step planning loop is solving a problem you don't have. Reaching for fine-tuning when better prompting (a clearer system prompt, better few-shot examples, structured output constraints) would achieve the same consistency is spending a training run and an ongoing retraining burden on something a prompt iteration cycle could have fixed in an afternoon. And reaching for RAG when the needed information is small, stable, and could just live directly in the system prompt (a short set of company policies that rarely change) adds a retrieval pipeline to solve a problem "put it in the prompt" already solves.
The discipline this lesson is really teaching is sequencing: prove the simplest approach can't hit the bar before adding the next layer of complexity, and be able to state in one sentence which specific gap justified each layer you did add.
A team ships a "summarize this single support ticket into three bullet points" feature by building a full agentic pipeline: the agent can search the wiki, look up account data, check three different internal tools, and finally summarize. In practice, 95% of tickets only need the ticket text itself to summarize well, and the search/lookup tools are rarely invoked. What's the diagnosis and the right fix?
The evidence (95% of tickets need only the ticket text) shows the task rarely requires multi-step decision-making or external lookups, meaning the defining condition for an agent architecture (deciding a sequence of actions from intermediate results) barely applies here. Maintaining a full agent for a task that's mostly a single-shot summarization adds unnecessary latency, failure surface, and engineering maintenance burden. The right fix is matching the architecture to the actual observed gap: a direct prompt for the common case, with a narrower, targeted RAG or tool-lookup path added only for the specific ticket types that genuinely need it -- not scaling the agent up further.