Phase 7: Multi-Model Pipeline Orchestration · 50 min · Python · Temporal

Error Handling & Step-Level Resume

Retry the failed step, not the entire pipeline. Resume from the last successful checkpoint. That's production-grade error handling.

Hiring signal: Step-level error handling and pipeline resume capability is an advanced production engineering skill — it shows you've operated generative media systems at scale.

What you will learn

Implement step-level retry: retry individual failed steps, not the entire pipeline
Implement partial failure recovery: resume from last successful step
Design idempotent pipelines: same input always produces same output (deterministic seeds)
Classify errors: transient (retry), content policy (route to different provider), quality (regenerate), fatal (alert human)

The Problem

A 9-step pipeline fails at step 7 (post-production). Without step-level resume:

Re-run the entire pipeline from step 1
Re-pay for steps 1-6 ($0.70 in API costs)
Wait 120s for steps 1-6 to re-execute
If step 7 fails again, repeat

With step-level resume:

Load checkpoint — steps 1-6 are already completed
Retry only step 7
Cost: $0 (steps 1-6 already done)
Time: 10s (only step 7)

Step-level error handling and resume is an advanced production skill — it shows you've operated generative media systems at scale.

What you'll build
Implement step-level retry (retry individual failed steps), partial failure recovery (resume from last successful checkpoint), idempotent pipelines (deterministic seeds), and error classification (transient, content policy, quality, fatal).

Step-Level Retry

Retry Individual Steps

import asyncio
import time
from dataclasses import dataclass, field

@dataclass
class StepResult:
    """Result of a single pipeline step."""
    step_id: str
    status: str = "pending"  # pending, running, completed, failed
    output: str = ""
    error: str = ""
    attempts: int = 0
    cost: float = 0.0

async def execute_with_retry(step_id: str, step_fn, inputs: dict,
                              max_retries: int = 3,
                              backoff_base: float = 2.0) -> StepResult:
    """Execute a step with exponential backoff retry."""
    result = StepResult(step_id=step_id)
    
    for attempt in range(1, max_retries + 1):
        result.attempts = attempt
        result.status = "running"
        
        try:
            output = await step_fn(**inputs)
            result.status = "completed"
            result.output = output
            return result
            
        except Exception as e:
            result.error = str(e)
            error_type = classify_error(e)
            
            if error_type == "fatal":
                # Don't retry fatal errors
                result.status = "failed"
                return result
            
            if attempt < max_retries:
                wait = backoff_base ** attempt
                print(f"  Step {step_id} attempt {attempt} failed: {e}")
                print(f"  Retrying in {wait}s... (error type: {error_type})")
                await asyncio.sleep(wait)
            else:
                result.status = "failed"
    
    return result

Exponential Backoff

# Attempt 1: fail → wait 2s
# Attempt 2: fail → wait 4s
# Attempt 3: fail → wait 8s
# Attempt 4: give up

backoff = [2**i for i in range(max_retries)]
# [1, 2, 4, 8, 16] for 5 retries

Why is exponential backoff used for retrying failed API calls?

It produces better quality output

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Error Classification, Pipeline Resume, Idempotent Pipelines, Error Handling Strategies by Type, Key Takeaways, What's Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.