Phase 3: Scale, Failure & Evaluation · ~75 minutes · Python (stdlib · `asyncio` · `sqlite3`)

Production Scaling — Queues, Checkpoints, Durability

Scaling multi-agent systems to thousands of concurrent runs requires durable execution — work queues plus checkpoints, so any worker can resume any run after any crash, provided lease handling, idempotent side effects, and deterministic replay are in place.

Hiring signal: Can operate production scaling in production

Introduction

LangGraph's runtime is the reference example: it writes a checkpoint after each super-step keyed by thread_id (Postgres by default); worker crashes release a lease and another worker resumes. Agents can sleep indefinitely waiting for human input. MegaAgent (arXiv:2408.09955) ran a per-agent producer-consumer queue with three states (Idle / Processing / Response) and two-layer coordination (intra-group chat + inter-group admin chat). Fiber/async beats thread-per-job for LLM streaming: threads sit idle 99% of the time waiting for tokens, fibers cooperatively yield on I/O. Counterpoint: Ashpreet Bedi's "Scaling Agentic Software" argues for FastAPI + Postgres + nothing else until load proves otherwise — simple architectures go further than expected. This lesson builds a durable checkpoint log, a per-agent work queue with state transitions, an async-vs-thread demo, and lands the pragmatic "start simple" rule.

Type: Learn + Build Languages: Python (stdlib, asyncio, sqlite3) Prerequisites: Phase 16 · 09 (Parallel Swarm Networks), Phase 16 · 13 (Shared Memory) Time: ~75 minutes

Objective

Learning objectives

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Agent