Build an AI Content Factory · 30 min · Python · httpx · BeautifulSoup4

Build the Input Preprocessors

The generators can only be as good as the text you hand them — messy input in, generic output out, no matter how good the Voice Profile is.

Hiring signal: Real-world input is never clean. Extracting usable text from a random article URL or a YouTube link people actually sent you is the unglamorous 80% of any content-repurposing tool that determines whether the other 20% (generation) is worth anything.

What you will learn

Extract clean article text from a URL
Extract a YouTube video's transcript from its video ID
Handle the realistic failure cases: missing transcripts, blocked pages, malformed input

Introduction

Three input types, three extraction problems: a URL might be a clean article or a paywalled mess, a YouTube video might have a transcript or might not, and raw text just needs light cleanup. Today you build all three, tested against real inputs you actually want to repurpose — not sanitized example URLs that always work.

Article extraction: fetch, then find the actual content

import httpx
from bs4 import BeautifulSoup

def extract_article(url: str) -> str:
    resp = httpx.get(url, timeout=10, follow_redirects=True,
                      headers={"User-Agent": "Mozilla/5.0"})
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "html.parser")
    for tag in soup(["script", "style", "nav", "footer", "header"]):
        tag.decompose()
    paragraphs = soup.find_all("p")
    return "\n\n".join(p.get_text(strip=True) for p in paragraphs)

Two things matter here that a naive soup.get_text() would miss: stripping script/style/nav/footer tags first (otherwise you get navigation menus and cookie-banner text mixed into your "article"), and pulling text specifically from <p> tags rather than the whole page body. Neither is bulletproof -- some sites structure content differently, and this is exactly the kind of thing worth testing against 3 real URLs, not 1, before you trust it.

Some pages will just fail, and that's not a bug to fix today
Paywalled articles, JavaScript-rendered sites that need a real browser to show content, and sites that actively block non-browser user agents will all produce empty or garbage output from this approach. A production-grade scraper would reach for a headless browser or a service built for this; for this course, the correct move is to detect the failure (empty or suspiciously short extracted text) and return a clear error rather than silently feeding 40 characters of nav-menu text into a generator that will then hallucinate a blog post out of nothing.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers YouTube transcripts: a newer API surface worth knowing about, What You're Building — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.