The Problem
"Compare our Q3 revenue from the North American and European markets and explain which region's growth was driven by new customer acquisition vs. expansion revenue." Drop this into a standard RAG system and watch it fail gracefully: the embedding of the full question retrieves semi-relevant documents, the model gets a muddy context, and the answer is a confident-sounding hedge.
The reason is structural. This question contains at least four distinct information needs: Q3 North America revenue, Q3 Europe revenue, North America acquisition vs. expansion breakdown, and Europe acquisition vs. expansion breakdown. These come from different tables, potentially different source systems, and the answer to each one might come from a different place in your data infrastructure. A single semantic search query cannot be simultaneously well-targeted for all four needs.
Query decomposition is the architectural fix. Instead of embedding the entire question and hoping the retrieval system understands its composite structure, you use an LLM to break the question into atomic sub-queries — each one narrow enough to be answered by a focused retrieval call. Each sub-query gets routed to the data source best suited to answer it. The results come back and get synthesized into a single coherent answer with proper attribution.
This is not a niche optimization. Any enterprise knowledge assistant that needs to handle analytical, comparative, or multi-faceted questions requires some version of query decomposition. The systems at Salesforce, ServiceNow, and Workday that answer business questions across CRM data, support tickets, and documentation all implement this pattern — they just call it different things.
Sub-query dependency tracking
Not all sub-queries can run in parallel. Sequential decomposition produces sub-queries where later ones depend on the results of earlier ones — you can't ask "what was the return rate for our top 3 products?" until you know which products ARE the top 3. Dependency tracking in the decomposition output is what separates a correct implementation from one that will fail on multi-hop questions.
Query decomposition patterns
The LLM decomposer produces a structured output that specifies not just the sub-queries but how they relate to each other. A robust schema:
{
"sub_queries": [
{
"id": "sq1",
"query": "Top 3 products by revenue in Q3",
"source": "sql_analytics",
"depends_on": null
},
{
"id": "sq2",
"query": "Return rate for {sq1.result[0]} in Q3 vs Q2",
"source": "sql_analytics",
"depends_on": "sq1"
},
{
"id": "sq3",
"query": "Return rate for {sq1.result[1]} in Q3 vs Q2",
"source": "sql_analytics",
"depends_on": "sq1"
}
]
}
Three decomposition patterns appear in practice:
Parallel decomposition: sub-queries are independent and can run simultaneously. "What are the prices of products A, B, and C?" decomposes into three independent lookups. Run with asyncio.gather() to minimize total latency.
Sequential (multi-hop) decomposition: later sub-queries depend on earlier results. "Compare performance of our top product against last year's top product" requires knowing the current top product before you can look up last year's comparator. Run sequentially; template the dependent query with the upstream result.
Hierarchical decomposition: a broad question breaks into sub-topics, each of which may break further. Useful for research-style questions. Treat it as a tree traversal with breadth-first execution where possible.
Semantic routing
A semantic router maps each sub-query to the right data source. Implementation options from cheapest to most accurate:
| Approach | Mechanism | Latency | Accuracy |
|---|
| Keyword routing | regex/keyword matching | <1ms | ~70% |
| Embedding-based routing | cosine similarity to source descriptions | ~10ms | ~88% |
| LLM-based routing | ask the model to classify | ~200ms | ~95% |
For production, embedding-based routing hits the sweet spot: fast enough to add negligible overhead, accurate enough to route correctly the vast majority of the time. The implementation: embed short descriptions of each source ("structured order and transaction data in SQL", "product documentation and user guides in vector DB", "real-time pricing and inventory via API"), then route each sub-query by embedding it and finding the nearest source description.
class SemanticRouter:
def __init__(self, sources: dict[str, str]):
# sources = {"sql": "structured order data...", "vector_db": "product docs..."}
self.source_embeddings = {
name: embed(description)
for name, description in sources.items()
}
def route(self, query: str) -> str:
query_embedding = embed(query)
scores = {
name: cosine_similarity(query_embedding, emb)
for name, emb in self.source_embeddings.items()
}
return max(scores, key=scores.get)
A user asks: "What were our top 3 products by revenue in Q3, and how did each one's return rate compare to Q2?" How should a decomposition agent break this down?
This is a multi-hop sequential decomposition. You cannot ask for the return rates of the top 3 products until you know WHICH products are the top 3 — query 1 blocks queries 2–4. Making this dependency explicit in the decomposition ensures the executor runs query 1 first, then templates its results into queries 2–4. Routing to SQL for all steps is correct — this is structured numeric data, not document retrieval.
Combining decomposed results
After running N sub-queries, you have N result sets with potentially different formats — SQL rows, retrieved documents, API responses. The synthesis step passes everything to the model with explicit labeling:
synthesis_prompt = f"""
Original question: {original_question}
Retrieved information:
[Sub-query 1: Top 3 products by Q3 revenue]
{sq1_results}
[Sub-query 2: Q3 vs Q2 return rate — Product A]
{sq2_results}
[Sub-query 3: Q3 vs Q2 return rate — Product B]
{sq3_results}
[Sub-query 4: Q3 vs Q2 return rate — Product C]
{sq4_results}
Synthesize a complete answer to the original question. Cite specific data points.
"""
Key synthesis considerations:
- Label each result set by the sub-query that produced it — the model needs to know what each block of information is for
- Handle conflicting results explicitly: if sub-query 2 and sub-query 3 give different return rate figures for the same product from different sources, instruct the model to flag the discrepancy rather than picking one silently
- Source attribution: include
source and date metadata in each result block so the model can cite them in the final answer
You have 3 data sources: a vector DB of support docs, a SQL database of order history, and a live web search API. A user asks "What's the current price of product X and has anyone complained about it?" What's the correct routing?
Different sub-queries have different optimal sources. Current price is structured data with a single correct value — SQL. Complaints are in unstructured support documents — vector DB semantic search. Web search is a fallback for real-time data not in internal systems. Forcing both through vector DB gives lower accuracy on the price question (documents about prices vs. the actual price record), and forces web search to have complete complaint history it likely doesn't.