Phase 3: Agentic RAG · 65 min · Python · Anthropic SDK · ChromaDB
Query Decomposition & Routing
A complex question is several simple questions wearing a trench coat — decompose it, route each piece to the right source, then synthesize.
Hiring signal: Query decomposition separates 'I built a RAG chatbot' from 'I built a production knowledge system.' Enterprise AI applications at Salesforce, ServiceNow, and Workday use query decomposition for complex analytical queries — knowing how to design and implement it is a concrete differentiator.
What you will learn
- Implement query decomposition: break complex questions into atomic sub-queries with dependency tracking
- Build a semantic router that directs each sub-query to the correct retrieval source
- Combine multiple retrieval results into a coherent final answer with source attribution
The Problem
"Compare our Q3 revenue from the North American and European markets and explain which region's growth was driven by new customer acquisition vs. expansion revenue." Drop this into a standard RAG system and watch it fail gracefully: the embedding of the full question retrieves semi-relevant documents, the model gets a muddy context, and the answer is a confident-sounding hedge.
The reason is structural. This question contains at least four distinct information needs: Q3 North America revenue, Q3 Europe revenue, North America acquisition vs. expansion breakdown, and Europe acquisition vs. expansion breakdown. These come from different tables, potentially different source systems, and the answer to each one might come from a different place in your data infrastructure. A single semantic search query cannot be simultaneously well-targeted for all four needs.
Query decomposition is the architectural fix. Instead of embedding the entire question and hoping the retrieval system understands its composite structure, you use an LLM to break the question into atomic sub-queries — each one narrow enough to be answered by a focused retrieval call. Each sub-query gets routed to the data source best suited to answer it. The results come back and get synthesized into a single coherent answer with proper attribution.
This is not a niche optimization. Any enterprise knowledge assistant that needs to handle analytical, comparative, or multi-faceted questions requires some version of query decomposition. The systems at Salesforce, ServiceNow, and Workday that answer business questions across CRM data, support tickets, and documentation all implement this pattern — they just call it different things.
Sub-query dependency tracking
Not all sub-queries can run in parallel. Sequential decomposition produces sub-queries where later ones depend on the results of earlier ones — you can't ask "what was the return rate for our top 3 products?" until you know which products ARE the top 3. Dependency tracking in the decomposition output is what separates a correct implementation from one that will fail on multi-hop questions.
Query Decomposition Patterns
The LLM decomposer produces a structured output that specifies not just the sub-queries but how they relate to each other. A robust schema:
{
"sub_queries": [
{
"id": "sq1",
"query": "Top 3 products by revenue in Q3",
"source": "sql_analytics",
"depends_on": null
},
{
"id": "sq2",
"query": "Return rate for {sq1.result[0]} in Q3 vs Q2",
"source": "sql_analytics",
"depends_on": "sq1"
},
{
"id": "sq3",
"query": "Return rate for {sq1.result[1]} in Q3 vs Q2",
"source": "sql_analytics",
"depends_on": "sq1"
}
]
}
Three decomposition patterns appear in practice:
Parallel decomposition: sub-queries are independent and can run simultaneously. "What are the prices of products A, B, and C?" decomposes into three independent lookups. Run with asyncio.gather() to minimize total latency.
Sequential (multi-hop) decomposition: later sub-queries depend on earlier results. "Compare performance of our top product against last year's top product" requires knowing the current top product before you can look up last year's comparator. Run sequentially; template the dependent query with the upstream result.
Hierarchical decomposition: a broad question breaks into sub-topics, each of which may break further. Useful for research-style questions. Treat it as a tree traversal with breadth-first execution where possible.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Semantic Routing, Combining Decomposed Results, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy