Phase 4: RAG Pipelines in Workflows · 50 min · Dify · n8n · Pinecone

RAG Fundamentals for Automation

Retrieval quality is the bottleneck, not model quality.

Hiring signal: RAG is the most common AI automation pattern after classification/extraction. Being able to explain why retrieval quality (not model quality) is the bottleneck in RAG systems, and how chunking and embeddings affect retrieval, demonstrates understanding that goes beyond 'I used the Dify knowledge base.'

What you will learn

Explain RAG in the context of workflow automation and why it differs from standalone RAG apps
Understand chunking fundamentals: chunk size, overlap, and their trade-offs
Understand embedding fundamentals: converting text to vectors, choosing embedding models
Identify why retrieval quality (not model quality) is the bottleneck in RAG automations

The Problem

A company builds a customer support chatbot. They upload 500 PDFs to a knowledge base and connect it to GPT-4o. When a customer asks "What's your return policy for opened items?", the bot responds with the general return policy — but misses the specific section about opened items that's on page 47 of a 200-page manual. The customer gets wrong information. The team blames the model and upgrades to GPT-4o. Same result. They switch to Claude 3.5 Sonnet. Same result.

The problem isn't the model. The problem is retrieval — the right information never reaches the model. The model can only answer based on what it's given. If retrieval returns the wrong chunks, the best model in the world will give the wrong answer.

RAG = Retrieve the right text + Generate an answer from it
RAG (Retrieval-Augmented Generation) has two parts: retrieval (find relevant text from your knowledge base) and generation (the LLM reads that text and answers). 90% of RAG problems are retrieval problems. If you retrieve the right text, even a small model gives good answers. If you retrieve the wrong text, no model can help.

The Concept

RAG in Automation vs Standalone Apps

Aspect	Standalone RAG App	RAG in Workflow Automation
Trigger	User types in chat	Webhook, email, schedule
Output	Chat response	Action: update CRM, send email, create ticket
Validation	User judges quality	System validates before acting
Fallback	"I don't know"	Route to human review, use template
Volume	One query at a time	Thousands per day
Cost sensitivity	Moderate	Critical (cost per execution)

The RAG Pipeline

┌──────────────────────────────────────────────────────────────┐
│  RAG PIPELINE                                                 │
│                                                              │
│  1. INGEST: Documents → chunks → embeddings → vector store  │
│                                                              │
│  2. QUERY: User question → embedding → vector search         │
│           → retrieve top-k relevant chunks                   │
│                                                              │
│  3. GENERATE: Question + retrieved chunks → LLM → answer    │
│                                                              │
│  4. ACT (automation): Answer → validate → route → action    │
└──────────────────────────────────────────────────────────────┘

Chunking: Splitting Documents

Chunking is how you split large documents into smaller pieces for retrieval:

Chunk Size	Tokens (~4 chars/token)	Best For	Trade-off
256 tokens	~1,000 chars	Precise matching, FAQs	May lose context
512 tokens	~2,000 chars	General purpose (recommended)	Balanced
1024 tokens	~4,000 chars	Complex topics needing context	May retrieve irrelevant parts
2048 tokens	~8,000 chars	Full sections, legal docs	Too much text per chunk

Overlap: Adjacent chunks share some text (typically 10–20% of chunk size) to avoid splitting important information at chunk boundaries.

Document: "The return policy allows 30 days for unopened items. For opened
items, returns are accepted within 14 days with a 20% restocking fee."

Chunk size: 512 tokens, overlap: 50 tokens

Chunk 1: "The return policy allows 30 days for unopened items. For opened
         items, returns are accepted within..."
Chunk 2: "...items, returns are accepted within 14 days with a 20% restocking fee."
         (overlap with chunk 1)

A customer asks "What's the restocking fee for opened items?" Your knowledge base has 500 documents. The answer is in one sentence on page 47 of a manual. What's the most likely reason the RAG system returns the wrong answer?

Retrieval failure is the most likely cause. The LLM can only answer from what it's given. If the chunk containing "20% restocking fee for opened items" wasn't in the top-k retrieved chunks, the LLM never sees it. Upgrading the model won't help — it can't read what wasn't retrieved. The fix is improving retrieval: better chunking, hybrid search, or higher top-k.

Embeddings: Converting Text to Vectors

Embeddings are numerical representations of text that capture semantic meaning:

"return policy"          → [0.12, -0.34, 0.56, ...] (1536 dimensions)
"refund process"         → [0.11, -0.31, 0.59, ...] (similar vector)
"shipping rates"         → [0.45, 0.22, -0.18, ...] (different vector)

Similar meanings → similar vectors. The vector store finds chunks whose vectors are closest to the query vector.

Embedding Model	Dimensions	Cost (per 1M tokens)	Best For
OpenAI text-embedding-3-small	1536	$0.02	Default, good balance
OpenAI text-embedding-3-large	3072	$0.13	High accuracy
Cohere embed-english-v3	1024	$0.10	English-only, efficient
Ollama nomic-embed-text	768	$0	Local, free

Why Retrieval Quality Is the Bottleneck

Problem	Model Upgrade Helps?	Retrieval Fix Helps?
Wrong information in answer	No — model can't use what it didn't retrieve	Yes — retrieve the right chunks
Hallucinated answer	Partially — better models hallucinate less	Yes — better context = less hallucination
Answer too generic	No — model is doing its best with generic chunks	Yes — retrieve more specific chunks
Missing answer ("I don't know")	No — model didn't find relevant info	Yes — improve retrieval coverage
Wrong tone or format	Yes — better prompting/model	No — retrieval doesn't affect tone

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Exercises, Key Terms, Common Pitfalls — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.