Objective
Article extraction: fetch, then find the actual content
import httpx
from bs4 import BeautifulSoup
def extract_article(url: str) -> str:
resp = httpx.get(url, timeout=10, follow_redirects=True,
headers={"User-Agent": "Mozilla/5.0"})
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")
for tag in soup(["script", "style", "nav", "footer", "header"]):
tag.decompose()
paragraphs = soup.find_all("p")
return "\n\n".join(p.get_text(strip=True) for p in paragraphs)
Two things matter here that a naive soup.get_text() would miss: stripping script/style/nav/footer tags first (otherwise you get navigation menus and cookie-banner text mixed into your "article"), and pulling text specifically from <p> tags rather than the whole page body. Neither is bulletproof -- some sites structure content differently, and this is exactly the kind of thing worth testing against 3 real URLs, not 1, before you trust it.
Some pages will just fail, and that's not a bug to fix today
Paywalled articles, JavaScript-rendered sites that need a real browser to show content, and sites that actively block non-browser user agents will all produce empty or garbage output from this approach. A production-grade scraper would reach for a headless browser or a service built for this; for this course, the correct move is to detect the failure (empty or suspiciously short extracted text) and return a clear error rather than silently feeding 40 characters of nav-menu text into a generator that will then hallucinate a blog post out of nothing.
YouTube transcripts: a newer API surface worth knowing about
from youtube_transcript_api import YouTubeTranscriptApi
def extract_youtube_transcript(video_id: str) -> str:
ytt_api = YouTubeTranscriptApi()
fetched = ytt_api.fetch(video_id)
return " ".join(snippet.text for snippet in fetched)
Note the instance-based pattern here (YouTubeTranscriptApi() then .fetch(...)) -- this library's API changed from an older static-method style (YouTubeTranscriptApi.get_transcript(...)) as of its 1.x releases. If you find older tutorials showing the static form, they're describing a previous major version; verify against the library's current README before trusting a code snippet you didn't write yourself, exactly the "APIs change without warning" caution this course keeps coming back to.
Not every video has a transcript, and that's not an edge case -- it's common
Auto-generated captions, disabled captions, and foreign-language-only tracks all mean fetch() can raise. Wrap the call, and when it fails, don't silently return an empty string -- return something explicit enough that the orchestrator (Lesson 4) can tell the difference between "this failed" and "this succeeded with genuinely no content."
You extract a YouTube transcript successfully, but it's 8,000 words from a 45-minute video. You feed the entire thing into a Twitter-thread generator prompt alongside the Voice Profile. What's the most likely problem?
It will probably still work, but a Twitter thread doesn't need 8,000 words of source material to extract 3-5 key points from. This is worth flagging now even though the fix (summarizing or truncating long transcripts before generation) is really Lesson 3's problem -- the preprocessor's job is to get clean text out; deciding how much of it each generator actually needs comes next.
What you're building
preprocessors.py: an article extractor, a YouTube transcript extractor, and a raw-text normalizer (light cleanup: strip excess whitespace, normalize encoding), each tested against at least 2 real inputs including one that's expected to fail — proving the failure path returns something explicit, not silent empty output.