Phase 4: RAG Pipelines in Workflows · 55 min · Cohere Rerank · Jina Reranker · Dify
The Problem
Your RAG system uses pure vector search. It works 80% of the time. But there's a pattern: when users search for specific terms like "ISO 27001" or "BOL-123456", vector search returns semantically similar but wrong chunks. The user searched for a specific standard number and got chunks about "security frameworks" — relevant conceptually, but not the exact document they need.
This is the limitation of pure vector search: it captures semantic similarity but misses exact keyword matches. The production solution is hybrid search (combine vector + keyword) and reranking (re-order candidates by true relevance). These two techniques typically improve retrieval accuracy by 15–30% over pure vector search.
Hybrid search catches what vector search misses
Vector search finds "conceptually similar" text. Keyword search finds "exact term matches." Hybrid search combines both: vector for conceptual queries ("how do returns work?"), keyword for specific queries ("ISO 27001", "BOL-123456", "SKU-7890"). In production, hybrid is the default — pure vector search is the exception.
The Concept
Why Pure Vector Search Fails
Query: "What does ISO 27001 require for access control?"
Vector search results (top-3):
1. [0.89] "Security frameworks like NIST and SOC 2 require access controls..."
→ Semantically similar, but wrong standard
2. [0.87] "Access control policies should include role-based permissions..."
→ Conceptually relevant, but generic
3. [0.85] "ISO 27001 is an international security standard..."
→ Right document, but lower vector score
Keyword search results (top-3):
1. "ISO 27001 Annex A.9: Access control — control objectives..."
→ Exact match on "ISO 27001" + "access control"
2. "ISO 27001 Clause 9.4: User access management..."
→ Exact match, highly relevant
3. "ISO 27001 requirements include access control, cryptography..."
→ Exact match, relevant
Hybrid search results (combined):
1. "ISO 27001 Annex A.9: Access control — control objectives..."
→ Best of both: exact keyword + semantic relevance
2. "ISO 27001 Clause 9.4: User access management..."
3. "ISO 27001 requirements include access control, cryptography..."
Hybrid Search Architecture
┌──────────────────────────────────────────────────────────────┐
│ HYBRID SEARCH PIPELINE │
│ │
│ [Query] ──→ [Embed] ──→ [Vector Search] ──→ Top-K candidates│
│ └→ [Tokenize] → [Keyword Search] → Top-K candidates │
│ │
│ [Combine & Re-rank] │
│ ├── Merge results (deduplicate) │
│ ├── Reciprocal Rank Fusion (RRF) or weighted scoring │
│ └── Return final top-K │
└──────────────────────────────────────────────────────────────┘
Reciprocal Rank Fusion (RRF)
RRF combines rankings from multiple search methods:
RRF score = Σ (1 / (k + rank_i))
Where:
k = constant (typically 60)
rank_i = rank of document in search method i
Example:
Document A: rank 1 in vector search, rank 5 in keyword search
RRF = 1/(60+1) + 1/(60+5) = 0.0164 + 0.0154 = 0.0318
Document B: rank 3 in vector search, rank 1 in keyword search
RRF = 1/(60+3) + 1/(60+1) = 0.0159 + 0.0164 = 0.0323
Document B ranks higher (appeared near top in both methods)
Reranking: The Second Pass
Reranking takes the top-K candidates from retrieval and re-scores them with a cross-encoder model:
[Query + 20 candidate chunks] → [Reranker model] → [Re-ordered top-5]
Reranker (Cohere Rerank / Jina Reranker):
- Cross-encoder: reads query AND chunk together
- Scores actual relevance (not just similarity)
- More accurate than vector similarity, but slower
- Used as second pass on 20-50 candidates → return top 5
| Model | Type | Latency | Cost | Best For |
|---|
| Cohere Rerank 3 | Cross-encoder | ~200ms | $2/1K searches | Production default |
| Jina Reranker v2 | Cross-encoder | ~150ms | Free tier, then paid | Cost-sensitive |
| bge-reranker | Open-source | Varies | $0 (self-hosted) | Full control |
Your hybrid search retrieves 20 candidate chunks for a query. Without reranking, the relevant chunk is at position 7. What happens if you only take top-5?
Without reranking, taking only top-5 means the relevant chunk at position 7 is missed. The LLM never sees it and can't answer correctly. The fix: either increase top-k to 10+ (more context but higher cost) or add reranking (re-order so the relevant chunk moves into top-5). Reranking is the better solution — it evaluates true relevance, moving position-7 chunks to position-2.
Parent-Child Chunking
TRADITIONAL CHUNKING:
Document → [chunk1: 500 tokens] [chunk2: 500 tokens] [chunk3: 500 tokens]
Search: match on chunk → return chunk
Problem: chunk may be too small to provide full context
PARENT-CHILD CHUNKING:
Document → Parent chunks (1024 tokens)
→ Each parent split into child chunks (256 tokens)
Search: match on CHILD chunks (precise, small)
Return: PARENT chunks (full context, larger)
Example:
Parent: "The return policy allows 30 days for unopened items. For opened
items, returns are accepted within 14 days with a 20% restocking
fee. Refunds are processed within 5-7 business days."
Child 1: "The return policy allows 30 days for unopened items."
Child 2: "For opened items, returns are accepted within 14 days with a
20% restocking fee."
Child 3: "Refunds are processed within 5-7 business days."
Query: "What's the restocking fee?"
Match: Child 2 (contains "restocking fee")
Return: Parent chunk (full return policy context)
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Exercises, Key Terms, Common Pitfalls — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.