Phase 4: Training LLMs · ~60 minutes · Python
Jamba — Hybrid SSM-Transformer
State space models (SSMs) and transformers want different things. Transformers buy quality via attention at quadratic cost. SSMs buy linear-time inference and constant memory via a recurrence but lag quality. AI21's Jamba (March 2024) and Jamba 1.5 (August 2024) put them in the same model: 1 Transformer layer for every 7 Mamba layers, MoE on every other block, and a 256k context window that fits on a single 80GB GPU. Mamba-3 (ICLR 2026) tightens the SSM side with complex-valued state spaces and MIMO projections. This lesson reads both architectures end to end and explains why the hybrid recipe has survived three years of scaling when pure-SSM and pure-Transformer long-context attempts have not.
What you will learn
- Explain the three primitives in a Jamba block — Transformer layers, Mamba layers, MoE — and the 1:7:even interleaving recipe.
- State what an SSM's recurrence looks like at a high level and why it enables constant-memory inference.
- Compute the KV cache footprint of a Jamba model at 256k context and compare to what a pure-Transformer model would need.
- Name the three Mamba-3 innovations (exponential-trapezoidal discretization, complex-valued state update, MIMO) and the problem each one targets.
Introduction
- Type: Learn
- Languages: Python (stdlib, layer-mix calculator)
- Prerequisites: Open Models: Architecture Walkthroughs, Native Sparse Attention (DeepSeek NSA)
- Time: ~60 minutes
Objective
Learning objectives
- Explain the three primitives in a Jamba block — Transformer layers, Mamba layers, MoE — and the 1:7:even interleaving recipe.
- State what an SSM's recurrence looks like at a high level and why it enables constant-memory inference.
- Compute the KV cache footprint of a Jamba model at 256k context and compare to what a pure-Transformer model would need.
- Name the three Mamba-3 innovations (exponential-trapezoidal discretization, complex-valued state update, MIMO) and the problem each one targets.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy