Phase 3: AI Document Processing & Data Extraction · 50 min · pdfplumber · PyPDF2 · unstructured.io

PDF Processing Pipelines — From Inbox to Structured Data

Every PDF is a pipeline; every pipeline needs error handling.

Hiring signal: PDF processing is the most common document automation use case. Being able to describe the full pipeline — email attachment extraction, text extraction methods (pdfplumber vs PyPDF2 vs unstructured.io), LLM field extraction, and multi-page handling — demonstrates real document automation experience.

What you will learn

Build a complete PDF processing pipeline: email → attachment → text extraction → LLM extraction → database
Compare text extraction tools: pdfplumber, PyPDF2, unstructured.io, LLM-based extraction
Design extraction prompts that tell the model what to find, not where to find it
Handle multi-page documents: chunking, per-page extraction, and aggregation

The Problem

An accounts payable team receives 500 invoices per day as PDF email attachments from 80 different vendors. Each vendor uses a different invoice template. The team manually enters each invoice into their ERP system: vendor name, invoice number, date, line items, total, tax, payment terms. It takes 4 minutes per invoice — 33 hours/day of data entry.

They need a pipeline: email arrives → extract PDF → extract text → LLM extracts fields → validate → enter in ERP. But PDFs are tricky: some are digital (text embedded), some are scanned (image only), some have tables that extraction tools mangle, and some are 50 pages long. Building a reliable pipeline requires choosing the right extraction tool for each PDF type and handling edge cases.

Tell the model what to find, not where to find it
Rules-based extraction says "the invoice number is at position (x, y)" or "it's the text after 'Invoice #'". LLM extraction says "find the invoice number" — the model figures out where it is. This is the fundamental shift that makes LLM extraction handle layout variation.

The Concept

PDF Types and Extraction Strategies

┌─────────────────────────────────────────────────────────────┐
│  PDF TYPE DETECTION                                          │
│                                                             │
│  [Receive PDF] → [Try pdfplumber/PyPDF2]                   │
│                      ↓                                      │
│               Text extracted?                               │
│               ├── Yes → Digital PDF → Use extracted text    │
│               └── No  → Scanned PDF → OCR (Tesseract/cloud)│
│                                                             │
│  Special cases:                                             │
│  - PDF with tables → Use pdfplumber.extract_tables()       │
│  - PDF with forms → Use pdfplumber.extract_form_fields()   │
│  - 50+ pages → Chunk by page, extract per page             │
└─────────────────────────────────────────────────────────────┘

Text Extraction Tool Comparison

Tool	Best For	Strengths	Weaknesses
pdfplumber	Digital PDFs, tables	Table extraction, form fields, precise positioning	Slow for large PDFs
PyPDF2	Simple digital PDFs	Fast, lightweight	No table support, basic text only
unstructured.io	Mixed formats	Handles PDF, DOCX, HTML, images; preserves structure	Setup complexity, dependencies
Tesseract	Scanned PDFs	Free, open-source, 100+ languages	Needs image conversion first
LLM vision	Complex layouts	GPT-4o can read PDFs directly, understands layout	Higher cost, token limits

Extraction Prompt Design

BAD (position-based):
"The invoice number is in the top-right corner, after 'Invoice #'"

GOOD (semantic):
"Find the invoice number — it's a unique identifier for this invoice.
It may be labeled 'Invoice #', 'Inv No', 'Bill #', or just a number
near the top of the document. Return it as a string."

BAD (rigid format):
"Extract the date in MM/DD/YYYY format from line 3"

GOOD (flexible):
"Find the invoice date. It may be in various formats (MM/DD/YYYY,
DD/MM/YYYY, Month DD, YYYY). Return it in ISO format (YYYY-MM-DD)."

Your LLM extraction prompt says "Extract the total amount from the bottom of the invoice." A vendor sends an invoice where the total is in the middle, next to a "Balance Due" label. What happens?

The LLM understands "total amount" by meaning, not position. Even though the prompt says "bottom of the invoice," the LLM will look for the semantic concept of a total amount. However, the prompt is poorly designed — saying "bottom of the invoice" adds unnecessary constraints. Better prompt: "Find the total amount due on this invoice. It may be labeled 'Total', 'Amount Due', 'Balance Due', or similar."

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Exercises, Key Terms, Common Pitfalls — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Chunking