Phase 4: Multimodal Deployment · ~180 minutes · Python

Document and Diagram Understanding

Documents are not photos. A PDF, scientific paper, invoice, or handwritten form has layout, tables, diagrams, footnotes, headers, and semantic structure that plain image understanding cannot capture. The pre-VLM stack was a pipeline: Tesseract OCR + LayoutLMv3 + table-extraction heuristics. The VLM wave replaced that with OCR-free models — Donut (2022), Nougat (2023), DocLLM (2023) — that emit structured markup directly. By 2026 the frontier is just "feed the page image to Claude Opus 4.7 at 2576px native," and the structured-markup output comes for free. This lesson reads the three-era arc of document AI.

Hiring signal: Understanding of document and diagram understanding internals

What you will learn

Explain the three eras of document AI: OCR pipeline, OCR-free, VLM-native.
Describe LayoutLMv3's three input streams: text, layout (bbox), image patches, with unified masking.
Compare Donut (OCR-free, image → markup), Nougat (scientific paper → LaTeX), DocLLM (layout-aware generative), PaliGemma 2 (VLM-native).
Pick a document model for a new task (invoices, scientific papers, handwritten forms, Chinese receipts).

Introduction

Type: Build Languages: Python (stdlib, layout-aware document parser skeleton) Prerequisites: Phase 12 · 05 (LLaVA), Phase 5 (NLP) Time: ~180 minutes

Learning Objectives

Explain the three eras of document AI: OCR pipeline, OCR-free, VLM-native.
Describe LayoutLMv3's three input streams: text, layout (bbox), image patches, with unified masking.
Compare Donut (OCR-free, image → markup), Nougat (scientific paper → LaTeX), DocLLM (layout-aware generative), PaliGemma 2 (VLM-native).
Pick a document model for a new task (invoices, scientific papers, handwritten forms, Chinese receipts).

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.