Phase 8: Multi-Agent Systems & Platform Architecture · 45 min · Claude Code · Python
Long-Running Agent Systems
A real 22-incident study found 70% of silent agent failures were caught by a human looking at the output, not by tests. Design for failures that are loud, attributable, and boring instead of relying on that.
Hiring signal: Knowing that idle periods, not just active execution, are where long-running agent bugs hide — and testing for it deliberately — is the specific practice that separates production agent-runtime experience from theory.
What you will learn
- Explain why idle-time periods, not just active execution, are a distinct source of long-running agent bugs
- Simulate realistic idle-time delays in tests to catch bugs that only manifest after real waiting periods
- Interpret the 22-incident study's finding that most silent failures were caught by human observation, not automated tests
- Design for failures that are loud, attributable, and boring instead of relying primarily on human vigilance
Introduction
Long-Running Agent Systems
A team's approval-gated deployment agent works perfectly in every test — request approval, resume, deploy, done, all in under a second because the test approves instantly. In production, a real approval sits in someone's queue overnight, and when they approve it at 9am the next day, the resume step fails: the credential the agent obtained when it first requested approval expired after thirty minutes of inactivity, something no test ever exercised because no test ever actually waited thirty minutes, let alone eight hours. The bug was never in the active-execution logic — every fast-path test genuinely exercised that correctly. It was specifically in what happens during realistic idle time, a code path that only a test simulating real waiting would ever reach.
Idle time is a distinct thing to test
c12-05-3 covered checkpointing and recovery for long-running tasks; this lesson covers something adjacent but different: the specific behaviors that only manifest during idle periods — waiting for human approval, waiting for an external system, waiting for a rate limit to clear — as opposed to active execution. Credentials expire. Sessions time out. Cached data goes stale. None of these show up in a test that resumes a paused task one second after pausing it, because one second isn't idle time in any way that matters. Testing a long-running system properly means deliberately simulating realistic idle durations — fast-forwarding a simulated clock to represent the wait, not literally sleeping for hours — so the specific bugs that only appear after real waiting get exercised before a human discovers them at 9am.
Fast tests and idle-simulated tests catch different bugs
A fast test (resume immediately after pausing) proves the resume logic is correct in principle. It says nothing about whether anything time-dependent (a credential, a cache entry, a session) survives a realistic wait. Both tests are necessary; neither substitutes for the other, and a system that's only ever tested with fast resumes has a real, untested gap exactly where idle time actually happens in production.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers What a real 22-incident study found, Design for loud, attributable, boring failures, This is where the earlier weekly-ritual finding earns its place, Build It — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy