Phase 3: LLM Architecture · ~90 minutes · Python

Data Pipelines for Pre-Training

The model is a mirror. It reflects whatever data you feed it. Feed it garbage, it reflects garbage with perfect fluency.

What you will learn

Build a streaming data pipeline that tokenizes, chunks, shuffles, and batches terabytes of text without loading it all into memory
Implement data quality filters (deduplication, language detection, content filtering) used in real pre-training pipelines
Create fixed-length training sequences with proper attention masks and document boundary handling
Profile pipeline throughput to ensure the dataloader keeps up with GPU training speed

Introduction

Type: Build
Languages: Python
Prerequisites: Tokenizers: BPE, WordPiece, SentencePiece, Building a Tokenizer from Scratch
Time: ~90 minutes

Objective

Learning objectives

Build a streaming data pipeline that tokenizes, chunks, shuffles, and batches terabytes of text without loading it all into memory
Implement data quality filters (deduplication, language detection, content filtering) used in real pre-training pipelines
Create fixed-length training sequences with proper attention masks and document boundary handling
Profile pipeline throughput to ensure the dataloader keeps up with GPU training speed

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.