Phase 4: RAG Pipelines in Workflows · 55 min · Dify · n8n · Pinecone

Building Knowledge Bases in Dify and n8n

A good knowledge base is 80% of a good RAG system.

Hiring signal: Building and configuring knowledge bases in both Dify and n8n is a hands-on skill that automation roles test directly. Being able to compare Dify's built-in knowledge base management with n8n's vector store integration, and choose the right approach for a given use case, demonstrates practical RAG production experience.

What you will learn

Create Dify knowledge bases: file upload, web crawler, online drive, online documents
Configure Dify chunking modes: General, Parent-child, Q&A
Configure Dify index methods: High Quality (vector/full-text/hybrid) vs Economical
Build n8n vector store integration with Pinecone, Qdrant, Supabase pgvector

The Problem

You understand RAG fundamentals: chunking, embeddings, retrieval. Now you need to actually build a knowledge base. The two main platforms for this in automation workflows are Dify (AI-native, built-in knowledge base management) and n8n (workflow platform with vector store integrations). They take fundamentally different approaches:

Dify: Upload documents, configure chunking in the UI, manage everything visually. Best for teams who want knowledge base management without code.
n8n: Connect to external vector stores (Pinecone, Qdrant, Supabase pgvector), control everything via workflow nodes. Best for teams who want programmatic control.

Choosing the right approach and configuring it correctly determines whether your RAG system retrieves the right information or returns irrelevant chunks.

Dify manages the knowledge base for you; n8n gives you control
Dify handles chunking, embedding, indexing, and retrieval in its UI — you upload documents and configure settings. n8n connects to external vector stores — you build the ingestion and retrieval pipeline yourself. Dify is faster to set up; n8n is more flexible and integrates with any workflow.

The Concept

Dify Knowledge Base Architecture

┌──────────────────────────────────────────────────────────────┐
│  DIFY KNOWLEDGE BASE                                          │
│                                                              │
│  Data Sources:                                               │
│  ├── File Upload (PDF, TXT, MD, DOCX, CSV, XLSX)           │
│  ├── Web Crawler (URL → scrape → chunk)                     │
│  ├── Online Drive (Google Drive, Notion)                    │
│  └── Online Documents (URL to specific doc)                 │
│                                                              │
│  Chunking Modes:                                             │
│  ├── General (fixed-size chunks with overlap)               │
│  ├── Parent-Child (small chunks match, parent provides ctx) │
│  └── Q&A (each chunk is a question-answer pair)             │
│                                                              │
│  Index Methods:                                              │
│  ├── High Quality (vector / full-text / hybrid)             │
│  │   ├── Vector: semantic search via embeddings             │
│  │   ├── Full-text: keyword search (BM25-style)             │
│  │   └── Hybrid: combine vector + full-text                 │
│  └── Economical (keyword index only, no embeddings)         │
│                                                              │
│  Retrieval Settings:                                         │
│  ├── Top-K (how many chunks to retrieve)                    │
│  ├── Score threshold (minimum similarity score)             │
│  └── Rerank (optional: Cohere/Jina reranker)                │
└──────────────────────────────────────────────────────────────┘

n8n Vector Store Integration

┌──────────────────────────────────────────────────────────────┐
│  n8n VECTOR STORE PIPELINE                                    │
│                                                              │
│  INGESTION WORKFLOW:                                         │
│  [File Trigger] → [Code: Extract Text] → [Code: Chunk]      │
│  → [Embeddings Node] → [Vector Store: Upsert]               │
│                                                              │
│  QUERY WORKFLOW:                                             │
│  [Webhook] → [Code: Prepare Query] → [Embeddings Node]      │
│  → [Vector Store: Query] → [AI: Generate Answer]            │
│                                                              │
│  Supported Vector Stores:                                    │
│  ├── Pinecone (managed, serverless)                         │
│  ├── Qdrant (open-source, self-hostable)                    │
│  ├── Supabase pgvector (Postgres extension)                 │
│  ├── Chroma (open-source, local)                            │
│  └── In-Memory (for testing)                                │
└──────────────────────────────────────────────────────────────┘

Dify vs n8n for Knowledge Bases

Feature	Dify	n8n
Setup speed	Fast (UI-guided)	Slower (build pipeline)
Chunking modes	General, Parent-Child, Q&A	Custom (Code node)
Index methods	Vector, Full-text, Hybrid	Depends on vector store
Reranking	Built-in option	Add as separate node
Data sources	File, web, drive, URL	Any (via workflow nodes)
Control level	Configuration options	Full programmatic control
Best for	AI apps, chatbots, quick RAG	Production workflows, custom pipelines
Cost	Dify cloud pricing	Vector store cost + API costs

You need to build a knowledge base from 200 PDFs and a website that updates daily. Which platform is better suited?

Dify is better suited because it handles both data sources natively: file upload for the 200 PDFs and web crawler for the daily-updating website. You configure the crawler schedule in Dify's UI. With n8n, you'd need to build a custom web scraper (HTTP + Code nodes), a PDF text extraction pipeline, and a chunking pipeline — more work for the same result. However, if you need custom chunking logic or want to integrate with a specific vector store, n8n gives more control.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Exercises, Key Terms, Common Pitfalls — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.