The Problem
Every production RAG system starts with the same tempting simplicity: user message comes in, retrieve top-k documents, append them to the prompt, call the LLM, return the answer. It works. Then you look at your telemetry and realize your agent is running a vector search for "What is 2+2?" It's spending 120ms and 1,000 tokens fetching documents about arithmetic that are completely irrelevant to a question the model knows from training.
Pipeline RAG treats retrieval as a preprocessing step applied unconditionally to every message. This is a reasonable default when you're prototyping, but it breaks down in production. A mixed-intent conversation — some factual questions the model knows, some company-specific questions that require retrieval, some complex multi-hop questions that require retrieval twice — needs a smarter approach.
Agentic RAG inverts the control flow. Instead of always retrieving before generating, the model decides at runtime whether to call the retrieval tool, what query to use, how many results to request, and whether to call it again if the first results are insufficient. The retrieval function becomes a tool the agent can invoke zero, one, or multiple times per user message — just like any other tool in its toolkit.
The result is a system that skips retrieval for questions it knows, does targeted single retrieval for straightforward lookup questions, and performs iterative multi-hop retrieval for complex questions that require reasoning across multiple documents. This isn't just an efficiency win: the model's control over retrieval also improves accuracy, because it can refine queries based on what it finds rather than being locked into the user's original phrasing.
Agentic RAG vs. Pipeline RAG
In pipeline RAG, retrieval is always step 1 — unconditional, every message. In agentic RAG, retrieval is an optional tool call — the model decides IF, WHEN, and HOW MANY TIMES to retrieve based on the question at hand. This single architectural change eliminates wasted retrievals and enables multi-hop reasoning.
Designing the retrieval tool
The retrieval tool schema is where the design work happens. A naive schema — search(query: str) -> list[str] — forces the agent to use a single text query with fixed parameters. A well-designed schema gives the agent meaningful control:
def search_knowledge_base(
query: str,
source: Literal["docs", "faq", "policies"] = "docs",
top_k: int = 5,
min_score: float = 0.7
) -> list[dict]:
"""
Search the knowledge base for relevant documents.
Args:
query: The search query — use targeted, specific language
source: Which corpus to search. Use 'faq' for common questions,
'policies' for rules and procedures, 'docs' for technical content
top_k: Number of results to return (1–10). Use fewer for simple questions.
min_score: Minimum cosine similarity threshold (0–1). Raise to 0.85 for
high-precision needs, lower to 0.6 if you expect sparse coverage.
Returns:
List of documents with keys: text, source, score, date, doc_id
"""
The source parameter lets the agent target the right corpus rather than cross-contaminating a policy question with technical documentation results. top_k control lets the agent request more results for complex multi-part questions and fewer for simple lookups, saving context tokens. min_score lets the agent raise the bar when it needs high-precision results — if the first call returns low-scoring documents, the agent can retry with a more specific query rather than reasoning from weakly relevant context.
Returning metadata alongside document text is not optional: the model needs score to judge relevance, date to assess freshness, and source to cite its answer. Without metadata, the model is reasoning blind about what it retrieved.
What is the key advantage of giving the agent control over top_k in the retrieval tool?
top_k control lets the agent match retrieval breadth to question complexity. A simple lookup ("what is our cancellation window?") needs 1–2 results. A multi-part question ("compare our return policy for US vs. international orders across all product categories") may need 8–10. Fixed top_k always over- or under-retrieves; agent-controlled top_k matches cost and context use to actual need.
Pipeline RAG vs. agentic RAG: A side-by-side flow
The architectural difference is visible in the call graph:
Pipeline RAG flow:
user_message
→ embed(user_message)
→ vector_search(embedding, top_k=5) ← always runs
→ build_prompt(message, retrieved_docs)
→ llm_call(prompt)
→ response
Agentic RAG flow:
user_message
→ llm_call(message, tools=[search_knowledge_base])
→ [if tool_use] search_knowledge_base(query, source, top_k)
→ [if results insufficient] search_knowledge_base(refined_query, ...)
→ llm_call(message, tool_results)
→ response
In pipeline RAG, the LLM never chooses whether to retrieve. In agentic RAG, the first LLM call decides: answer directly, call the tool once, or call it multiple times with progressively refined queries.
The extra LLM call in the agentic path is the cost of control. For applications where retrieval is genuinely needed for most messages, this overhead outweighs the benefits — pipeline RAG wins on pure latency. The crossover point is roughly when more than 30% of messages don't need retrieval, or when more than 20% of messages need multi-hop retrieval that a pipeline can't do at all.
A user asks your agentic RAG system: "What is 7 × 8?" The agent should:
This is agentic RAG's core efficiency advantage over pipeline RAG. The model knows 7 × 8 = 56 without retrieval. Pipeline RAG would still trigger a vector search — wasting ~100ms and ~1,000 tokens to retrieve arithmetic documents that don't help. The agentic agent correctly identifies this as a question it can answer directly. Over thousands of such questions in a mixed-intent application, this adds up to significant cost and latency savings.
Iterative query refinement
The most powerful agentic RAG behavior is iterative refinement — calling the retrieval tool multiple times, each time with a better query informed by what the previous call returned.
Scenario: a user asks "What is our refund policy for international orders placed during a promotional period?" The agent calls search_knowledge_base("refund policy", top_k=5) and gets 5 results about standard domestic refunds. None address international orders or promotional periods.
A pipeline system would answer from those 5 inadequate documents. An agentic system reasons: "My initial results don't cover the specific conditions in this question. I need more targeted queries." It calls:
search_knowledge_base("international orders refund policy", source="policies", top_k=3)search_knowledge_base("promotional period terms and conditions", source="policies", top_k=3)
It combines both result sets and generates a complete answer.
This multi-hop behavior is not possible in a pipeline — the pipeline has no mechanism to react to the quality of what it retrieved. The agent can, because it reads its tool results and decides what to do next.