Phase 8: Production Agent Infrastructure · 60 min · Python · asyncio · Anthropic SDK

Async Tool Execution & Parallelism

Parallel tool calls are the performance multiplier that makes agents fast enough for real users.

Hiring signal: Almost all production agent code is async. Engineers who understand the event loop, concurrency vs. parallelism in Python, and patterns like `gather()` vs. `as_completed()` are ready for production. This is tested in take-home assignments and system design rounds at companies running high-throughput agent systems.

What you will learn

Implement parallel tool execution with asyncio.gather() and asyncio.as_completed()
Apply timeout and cancellation patterns to prevent runaway async tasks
Measure the parallelism benefit: compare serial vs. parallel execution on a realistic task

The Problem

When an agent calls tools serially — one after another — it wastes time waiting. Each await blocks the next call from starting. For an agent that needs to call 6 tools, this means the user waits for the sum of all call durations. In real-world systems with network I/O, this adds up fast.

Async programming with Python's asyncio module is the performance multiplier that makes agents fast enough for real-time user interactions. Instead of waiting for each tool to return before starting the next, you fire them all at once and collect the results together.

asyncio Fundamentals for Tool Execution

The core pattern is simple: replace sequential await calls with asyncio.gather().

Serial execution — the slow way:

result1 = await tool_search_web(query)
result2 = await tool_query_database(user_id)
result3 = await tool_call_api(endpoint)
# Total time: sum of all three

Parallel execution — the fast way:

result1, result2, result3 = await asyncio.gather(
    tool_search_web(query),
    tool_query_database(user_id),
    tool_call_api(endpoint)
)
# Total time: max of all three

asyncio.gather() runs all coroutines concurrently — they're all in-flight at the same time. The wall-clock time equals the duration of the slowest call, not the sum. For 3 calls of 2 seconds each, that's 2 seconds instead of 6.

asyncio.as_completed() is the streaming alternative — it yields results as they arrive rather than waiting for all to finish:

tasks = [tool_search_web(q) for q in queries]
async for coro in asyncio.as_completed(tasks):
    result = await coro
    process_result(result)  # handle immediately, don't wait for others

Use gather() when you need all results before proceeding. Use as_completed() when you can start processing partial results — for example, display the first search result to the user while the others are still fetching.

Three API calls each take 2 seconds. Running them in parallel with asyncio.gather() takes:

asyncio.gather() runs all coroutines concurrently. Wall-clock time = max(individual times), not sum(individual times). All 3 API calls are in-flight simultaneously — the async event loop handles all 3 while awaiting each. This is the fundamental parallelism benefit: 3 independent I/O operations in 2 seconds instead of 6.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Timeouts and Cancellation, Dependency-Aware Parallel Execution, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Agent