Phase 4: Training LLMs · ~60 minutes · Python

Async and Hogwild! Inference

Speculative decoding (Phase 10 · 15) parallelizes tokens within one sequence. Multi-agent frameworks parallelize across whole sequences but force explicit coordination (voting, sub-task splitting). Hogwild! Inference (Rodionov et al., arXiv:2504.06261) does something else: run N instances of the same LLM in parallel against a SHARED key-value cache. Each worker sees every other worker's generated tokens instantly. Modern reasoning models — QwQ, DeepSeek-R1 — can self-coordinate through that shared cache without any fine-tuning. The approach is experimental but it opens an entirely new axis of inference parallelism that sits orthogonal to spec decode. This lesson implements a two-worker Hogwild! simulator in stdlib Python and explains why the shared-cache collaboration emerges from the existing model's reasoning abilities.

Hiring signal: Understanding of async and hogwild! inference internals

What you will learn

Describe the three common parallel-LLM topologies (voting, sub-task, Hogwild!) and name which problems each one targets.
State the core Hogwild! setup: multiple workers, one shared KV cache, emergent coordination via self-prompting.
Compute the wall-time speedup of Hogwild! as a function of worker count `N`, task-level parallelism `p`, and coordination overhead `c`.
Implement a two-worker Hogwild! simulator on a toy problem and observe the emergent task division.

Introduction

Type: Build Languages: Python (stdlib) Prerequisites: Phase 10 · 12 (inference optimization), Phase 10 · 15 (speculative decoding) Time: ~60 minutes

Learning Objectives

Describe the three common parallel-LLM topologies (voting, sub-task, Hogwild!) and name which problems each one targets.
State the core Hogwild! setup: multiple workers, one shared KV cache, emergent coordination via self-prompting.
Compute the wall-time speedup of Hogwild! as a function of worker count N, task-level parallelism p, and coordination overhead c.
Implement a two-worker Hogwild! simulator on a toy problem and observe the emergent task division.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build It, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.