Objective
Backup before you touch anything
import shutil
import difflib
def modify_file_safely(path: str, new_content: str) -> str:
backup_path = path + ".bak"
shutil.copy2(path, backup_path)
with open(path, "r", encoding="utf-8") as f:
old_content = f.read()
with open(path, "w", encoding="utf-8") as f:
f.write(new_content)
diff = "\n".join(difflib.unified_diff(
old_content.splitlines(), new_content.splitlines(),
fromfile=f"{path} (before)", tofile=f"{path} (after)", lineterm="",
))
return diff
The .bak copy exists so restore_from_backup (Lesson 5) has something to restore FROM if a later step in a multi-file plan fails — this is what makes rollback possible at all. The diff isn't optional polish either: it's the artifact that lets you (or the agent's own summary in Lesson 6) show exactly what changed, which matters enormously when you're trusting an agent's judgment about your own code.
This is the exact mistake the SPEC's hotspot list warns about
Writing open(path, "w").write(new_content) directly, with no backup step, works fine right up until a bug in the agent's generated code corrupts a file you have no way to recover — because you overwrote the only copy. The backup-before-write pattern above costs one extra line (shutil.copy2) and is the entire difference between "recoverable mistake" and "lost work." Do not skip this to move faster through the lesson.
Running tests as ground truth
import subprocess
def run_tests(repo_path: str) -> dict:
result = subprocess.run(
["python", "-m", "pytest", "-v", "--tb=short"],
cwd=repo_path, capture_output=True, text=True, timeout=60,
)
return {
"passed": result.returncode == 0,
"stdout": result.stdout,
"stderr": result.stderr,
"returncode": result.returncode,
}
--tb=short keeps failure tracebacks compact enough to feed back into the model without burning excess tokens on stack-trace noise, while still including the actual assertion failure. A timeout matters here too: a test suite that hangs (an infinite loop introduced by a bad generated change, for instance) would otherwise block the whole agent indefinitely.
Pytest's exact output format varies -- parse for signal, not exact strings
Different pytest versions and configurations format output slightly differently. Rather than regex-matching pytest's exact text, this course's fix loop passes returncode (0 = all passed, nonzero = something failed) as the hard signal, and the raw stdout/stderr as unstructured context for the model to read and reason about -- more robust than trying to parse pytest's output into a precise structured format that might shift between versions.
"Tests passed" only means what your test suite actually checks
run_tests returning passed: True means the existing suite didn't regress -- it does NOT mean a newly created file is correct. If the agent creates power_utils.py and nothing in tests/ imports or exercises it, the suite passes trivially, even if power_utils.py is broken. This is a real gap, not a hypothetical: a plan step's acceptance_criteria (e.g. "module imports cleanly") is descriptive metadata for the commit summary -- it is never independently checked by code. If you want a create step actually verified, the plan needs to include a step that adds a test importing the new file, so run_tests has something to fail on if it's wrong.
The fix loop: bounded, and honest when it gives up
Your fix loop tries 3 times to make a failing test pass, and on the 3rd attempt it still fails — a slightly different way each time. What should the agent do?
A hard cap on retries exists specifically so a stuck agent doesn't burn unbounded tokens chasing a fix that isn't converging, and stopping means genuinely stopping -- restoring the file to its pre-attempt state and surfacing the real failure, not leaving a broken intermediate version in place or, far worse, "fixing" the problem by deleting the test that caught it.
What you're building
execution_engine.py: modify_file_safely (backup + diff), run_tests (subprocess + structured pass/fail), and the fix loop (write → test → on failure, feed the real test output back to the model for a fix → retry, max 3 attempts, restore-and-report on final failure) — tested on a single-file plan step against your test repo, both a task that succeeds and one deliberately set up to exhaust all 3 retries.