Phase 2: Advanced RL · ~45 minutes · Python

Sim-to-Real Transfer

A policy trained in a simulator that fails on hardware is a policy that memorized the simulator. Domain randomization, domain adaptation, and system identification are the three tools to make learned controllers cross the reality gap.

Hiring signal: Understanding of sim-to-real transfer internals

What you will learn

Implement sim-to-real transfer from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Learn Languages: Python Prerequisites: Phase 9 · 08 (PPO), Phase 2 · 10 (Bias/Variance) Time: ~45 minutes

The Problem

Training a real robot is slow, dangerous, and expensive. A biped takes millions of training episodes to learn to walk; a real biped that falls over even once breaks hardware. Simulation gives you unlimited resets, deterministic reproducibility, parallel environments, and no physical damage.

But simulators are wrong. Bearings have more friction than MuJoCo models. Cameras have lens distortion the simulator does not include. Motors have delays, backlash, and saturation that 99% of sim models skip. Wind, dust, and variable lighting sabotage a policy trained on sterile rendering. The reality gap — systematic difference between sim distribution and real distribution — is the central problem of deployed RL for robotics.

You need a policy that is robust to sim-to-real distribution shift. Three historical approaches: randomize the simulator (domain randomization), adapt the policy with a little real data (domain adaptation / fine-tuning), or identify the real system's parameters and match them (system identification). In 2026 the dominant recipe combines all three with massive parallel simulation (Isaac Sim, Isaac Lab, Mujoco MJX on GPU).

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build It, Pitfalls, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.