Phase 1: Pretraining at Scale · ~90 minutes · Python

HDF5 Tokenized Corpus

The downloaded corpus has to land in a layout the trainer can stream from at line speed.

Hiring signal: Can build hdf5 tokenized corpus end to end

What you will learn

Stream documents into a resizable HDF5 integer dataset with deterministic chunking.
Shard the write across multiple HDF5 files so failure is bounded and parallelism is possible.
Read tokens back through HDF5's page-cache-backed chunked layout so the dataloader copies into batch buffers only at batch time.
Implement a sliding-window dataloader that emits fixed-length training sequences with explicit packing rules.

Introduction

JSONL on disk does not survive 16 dataloader workers. HDF5 with a resizable, chunked integer dataset does. This lesson builds streaming tokenization into a resizable HDF5 dataset, sharded write across multiple files, memory-mapped read at training time, and a sliding-window dataloader that produces fixed-length sequences with the right packing.

Type: Build Languages: Python Prerequisites: Phase 19 lessons 30-37 Time: ~90 minutes

Objective

Learning objectives

Stream documents into a resizable HDF5 integer dataset with deterministic chunking.
Shard the write across multiple HDF5 files so failure is bounded and parallelism is possible.
Read tokens back through HDF5's page-cache-backed chunked layout so the dataloader copies into batch buffers only at batch time.
Implement a sliding-window dataloader that emits fixed-length training sequences with explicit packing rules.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.