Phase 0: Platforms & Inference Economics · ~60 minutes · Python (stdlib · toy gateway-routing simulator)

AI Gateways — LiteLLM, Portkey, Kong AI Gateway, Bifrost

A gateway sits between your apps and model providers.

Hiring signal: Can operate ai gateways in production

What you will learn

Enumerate the six core gateway features (routing, fallback, retries, rate limits, secrets, observability, guardrails).
Map four 2026 gateways (LiteLLM, Portkey, Kong AI, Bifrost) to scale ceilings and use cases.
Cite the Kong benchmark (228% vs Portkey, 859% vs LiteLLM) and explain why it matters for >500 RPS.
Choose self-hosted vs managed given data residency and ops budget.

Introduction

Core features are provider routing, fallback, retries, rate limiting, secret references, observability, guardrails. Market split in 2026: LiteLLM is MIT OSS with 100+ providers, OpenAI-compatible, but breaks down around ~2000 RPS (8 GB memory, cascading failures in published benchmarks); best for Python, <500 RPS, dev/prototyping. Portkey is control-plane-positioned (guardrails, PII redaction, jailbreak detection, audit trails), went Apache 2.0 open-source March 2026, 20-40 ms latency overhead, $49/mo production tier. Kong AI Gateway built on Kong Gateway — Kong's own benchmark on same 12 CPUs: 228% faster than Portkey, 859% faster than LiteLLM; $100/model/month pricing (max 5 on Plus tier); enterprise-fit if you're already on Kong. Bifrost (Maxim AI) — automatic retries with configurable backoff, fallback to Anthropic on OpenAI 429. Cloudflare / Vercel AI Gateways — managed, zero-ops, basic retry. Data residency drives the self-host decision; Portkey and Kong sit in the middle with OSS + optional managed.

Type: Learn Languages: Python (stdlib, toy gateway-routing simulator) Prerequisites: Phase 17 · 01 (Managed LLM Platforms), Phase 17 · 16 (Model Routing) Time: ~60 minutes

Objective

Learning objectives

Enumerate the six core gateway features (routing, fallback, retries, rate limits, secrets, observability, guardrails).
Map four 2026 gateways (LiteLLM, Portkey, Kong AI, Bifrost) to scale ceilings and use cases.
Cite the Kong benchmark (228% vs Portkey, 859% vs LiteLLM) and explain why it matters for >500 RPS.
Choose self-hosted vs managed given data residency and ops budget.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Guardrails
Inference

Related lessons

Managed LLM Platforms — Bedrock, Vertex AI, Azure OpenAI
Inference Platform Economics — Fireworks, Together, Baseten, Modal, Replicate, Anyscale
Batch APIs — the 50% Discount as Industry Standard
Model Routing as a Cost-Reduction Primitive

Browse all courses · View pricing · DeVenture Academy