Phase 2: Autonomous Research Agent · ~90 minutes · Python

Experiment Runner

The loop is only as honest as its measurements. Build the runner that takes a spec, executes it in a sandboxed subprocess, and emits a json metrics blob the evaluator can trust.

Hiring signal: Can build experiment runner end to end

What you will learn

Encode an experiment as a typed spec the runner can serialise to a subprocess.
Launch a subprocess with a hard wall clock timeout and a soft memory cap, and surface both as terminal conditions.
Capture stdout, stderr, and the structured metrics blob into a single result record.
Build an ablation table that sweeps one configuration knob at a time over a fixed base spec.
Keep every result deterministic given a seed so the evaluator sees the same numbers across runs.

Introduction

Type: Build Languages: Python Prerequisites: Phase 19 Track A lessons 20-29 Time: ~90 minutes

Objective

Learning objectives

Encode an experiment as a typed spec the runner can serialise to a subprocess.
Launch a subprocess with a hard wall clock timeout and a soft memory cap, and surface both as terminal conditions.
Capture stdout, stderr, and the structured metrics blob into a single result record.
Build an ablation table that sweeps one configuration knob at a time over a fixed base spec.
Keep every result deterministic given a seed so the evaluator sees the same numbers across runs.

Why a subprocess

A research loop runs untrusted code. The hypothesis came from a sampler, the experiment script came from the same path; treating either as safe in-process is asking for a crash that takes the orchestrator down. Subprocesses are the simplest isolation the language ships: a separate process, an independent address space, a signal handle on the parent side.

The runner here does not implement full sandboxing. There is no cgroup, no seccomp filter, no namespace remapping. What it does have is a wall clock timeout, a polling loop for memory growth, and a kill path that terminates the process on either limit. That is the runtime contract every more elaborate sandbox extends. The lesson keeps the contract small enough to read in one sitting.

The ExperimentSpec shape

ExperimentSpec
  spec_id        : str            (stable id, "exp_001")
  hypothesis_id  : int            (link back to the queue from lesson 50)
  script_path    : str            (path to the python script to run)
  config         : dict           (passed to the script as one json arg)
  seed           : int            (deterministic seed for the experiment)
  wall_timeout_s : float          (hard timeout, killed on exceed)
  memory_cap_mb  : int            (soft cap, polled; killed on exceed)
  metric_keys    : list[str]      (which fields the evaluator will read)

The script lives on disk; the runner writes the config to a temp file path that the script reads. The script is expected to print a single json line on stdout whose keys are a superset of metric_keys. Anything else on stdout is captured but ignored by the metrics parser.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.