Phase 4: Training LLMs · ~60 minutes · Python

Jamba — Hybrid SSM-Transformer

State space models (SSMs) and transformers want different things. Transformers buy quality via attention at quadratic cost. SSMs buy linear-time inference and constant memory via a recurrence but lag quality. AI21's Jamba (March 2024) and Jamba 1.5 (August 2024) put them in the same model: 1 Transformer layer for every 7 Mamba layers, MoE on every other block, and a 256k context window that fits on a single 80GB GPU. Mamba-3 (ICLR 2026) tightens the SSM side with complex-valued state spaces and MIMO projections. This lesson reads both architectures end to end and explains why the hybrid recipe has survived three years of scaling when pure-SSM and pure-Transformer long-context attempts have not.

Hiring signal: Understanding of jamba — hybrid ssm-transformer internals

What you will learn

Explain the three primitives in a Jamba block — Transformer layers, Mamba layers, MoE — and the 1:7:even interleaving recipe.
State what an SSM's recurrence looks like at a high level and why it enables constant-memory inference.
Compute the KV cache footprint of a Jamba model at 256k context and compare to what a pure-Transformer model would need.
Name the three Mamba-3 innovations (exponential-trapezoidal discretization, complex-valued state update, MIMO) and the problem each one targets.

Introduction

Type: Learn Languages: Python (stdlib, layer-mix calculator) Prerequisites: Phase 10 · 14 (open-model architectures), Phase 10 · 17 (native sparse attention) Time: ~60 minutes

Learning Objectives

Explain the three primitives in a Jamba block — Transformer layers, Mamba layers, MoE — and the 1:7:even interleaving recipe.
State what an SSM's recurrence looks like at a high level and why it enables constant-memory inference.
Compute the KV cache footprint of a Jamba model at 256k context and compare to what a pure-Transformer model would need.
Name the three Mamba-3 innovations (exponential-trapezoidal discretization, complex-valued state update, MIMO) and the problem each one targets.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.