Phase 1: Retrieval & Knowledge Systems · 30 hours · Python (pipeline) · TypeScript (viewer UI)

Capstone 04 — Multimodal Document QA (Vision-First PDF, Tables, Charts)

The 2026 document-QA frontier moved away from OCR-then-text and toward vision-first late interaction.

Hiring signal: Can build capstone 04 end to end

Introduction

ColPali, ColQwen2.5, and ColQwen3-omni treat each PDF page as an image, embed it with multi-vector late interaction, and let the query attend to patches directly. On financial 10-Ks, scientific papers, and handwritten notes this pattern beats OCR-first by a large margin. Build the pipeline end to end on 10k pages and publish the side-by-side against OCR-then-text.

Phases exercised: P4 · P5 · P7 · P11 · P12 · P17

Type: Capstone Languages: Python (pipeline), TypeScript (viewer UI) Prerequisites: Phase 4 (computer vision), Phase 5 (NLP), Phase 7 (transformers), Phase 11 (LLM engineering), Phase 12 (multimodal), Phase 17 (infrastructure) Time: 30 hours

Objective

Learning objectives

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.