Phase 7: Multi-Model Pipeline Orchestration · 55 min · Python · fal.ai API · Replicate API

Model Routing & Fallback

Multi-provider routing, fallback chains, cost-aware routing, content policy routing — production pipelines never depend on a single provider.

Hiring signal: Model routing and fallback engineering is tested in system design interviews — 'design a video generation service with fallback providers' is a common question.

What you will learn

Build multi-provider routing: FLUX for quality, SDXL for cost — route based on use case
Implement fallback chains: Sora → Veo → Kling → fail gracefully
Implement cost-aware routing: cheap models for previews, expensive models for final output
Implement content policy routing: if one provider rejects prompt, try alternative provider

The Problem

A video generation service uses only Runway. Then:

Runway API goes down for 2 hours — all requests fail
A prompt gets rejected by Runway's content policy — user gets an error
The team needs cheaper generation for preview thumbnails — Runway is $0.50/video
Runway rate limit is hit during peak hours — requests queue up

Production pipelines need multi-provider routing with fallback chains.

What you'll build
Build a multi-provider router with fallback chains (Sora → Veo → Kling), cost-aware routing (cheap for previews, expensive for final), and content policy routing (if one provider rejects, try another).

Multi-Provider Routing

Provider Registry

from dataclasses import dataclass, field
from typing import Optional

@dataclass
class Provider:
    """A generation provider."""
    name: str
    model: str
    category: str  # "image", "video", "audio"
    cost: float  # per generation
    latency: float  # seconds
    quality_score: float  # 0-1
    rate_limit: int  # requests per minute
    api_type: str  # "sync", "async"
    supports_nsfw: bool = False

PROVIDERS = {
    "image": [
        Provider("fal-flux", "FLUX.1-dev", "image", 0.04, 15, 0.95, 100, "async"),
        Provider("fal-sdxl", "SDXL", "image", 0.02, 10, 0.85, 200, "async"),
        Provider("openai-dalle", "dall-e-3", "image", 0.04, 20, 0.90, 50, "sync"),
    ],
    "video": [
        Provider("runway", "gen-3", "video", 0.50, 60, 0.92, 20, "async"),
        Provider("kling", "kling-v1", "video", 0.35, 90, 0.88, 30, "async"),
        Provider("veo", "veo-2", "video", 0.40, 120, 0.90, 15, "async"),
    ],
    "audio": [
        Provider("elevenlabs", "multilingual-v2", "audio", 0.01, 5, 0.95, 100, "sync"),
        Provider("openai-tts", "tts-1", "audio", 0.005, 3, 0.85, 200, "sync"),
    ],
}

Router

class ModelRouter:
    """Multi-provider model router."""
    
    def __init__(self, providers: dict):
        self.providers = providers
    
    def select_provider(self, category: str, strategy: str = "quality") -> Provider:
        """Select a provider based on strategy."""
        available = self.providers.get(category, [])
        if not available:
            raise ValueError(f"No providers for category: {category}")
        
        if strategy == "quality":
            return max(available, key=lambda p: p.quality_score)
        elif strategy == "cost":
            return min(available, key=lambda p: p.cost)
        elif strategy == "speed":
            return min(available, key=lambda p: p.latency)
        elif strategy == "balanced":
            # Score = quality / cost (value for money)
            return max(available, key=lambda p: p.quality_score / p.cost)
        else:
            return available[0]

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Fallback Chains, Cost-Aware Routing, Content Policy Routing, Combined Router, Key Takeaways, What's Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.