Phase 1: Pretraining at Scale · ~90 minutes · Python

Large Corpus Downloader

Training a language model begins long before the first forward pass.

Hiring signal: Can build large corpus downloader end to end

What you will learn

Stream remote shards with `urllib` and decompress with `zstandard` without buffering the whole file in memory.
Resume partial downloads by issuing HTTP `Range` requests against a verified byte offset.
Build a MinHash signature per document and bucket it with LSH so near-duplicates collide.
Emit a shard manifest with content hash, byte size, document count, and dedup verdict.

Introduction

The corpus has to land on disk, decompressed, deduplicated, and addressable, with the resume story already worked out before the network drops at 4 percent. This lesson builds a streaming downloader that pulls compressed shards, decompresses on the fly with Zstandard, fingerprints near-duplicates via MinHash plus locality-sensitive hashing, and writes a shard manifest the rest of the pipeline can trust.

Type: Build Languages: Python Prerequisites: Phase 19 lessons 30-37 Time: ~90 minutes

Objective

Learning objectives

Stream remote shards with urllib and decompress with zstandard without buffering the whole file in memory.
Resume partial downloads by issuing HTTP Range requests against a verified byte offset.
Build a MinHash signature per document and bucket it with LSH so near-duplicates collide.
Emit a shard manifest with content hash, byte size, document count, and dedup verdict.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.