Phase 4: Harness Engineering · 45 min · Claude Code · Python

Feedback Loops and Self-Correction

The fix depends entirely on which category the failure actually is — 'switch to a better model' doesn't repair a missing document.

Hiring signal: Classifying failures by category (context, model, tool, spec, environment) before proposing a fix demonstrates the same diagnostic discipline this course has built since the sycophancy and debugging lessons — applied here at the system level instead of a single bug.

What you will learn

Classify an agent failure into one of five categories: context, model, tool, spec, environment
Explain why misclassifying a failure's category leads to a fix that doesn't address the real cause
Describe the outer-loop calibration pattern: monitoring failure patterns over time to adjust the harness itself
Build a failure classifier and outer-loop calibrator that recommends a harness change from repeated failure patterns

Introduction

Feedback Loops and Self-Correction

A team notices their agent's transcription-confidence flag has fired eleven times this week for the same podcast feed. Their first response is to upgrade to a more capable transcription model, on the theory that a smarter model would simply do better. Confidence scores don't improve. It turns out the actual problem is that this particular feed's audio files are encoded at an unusually low bitrate that the harness's context never surfaces to the transcription step — the model was never told, and no model, however capable, transcribes audio it can barely make out any better than a less capable one would. The team spent a week and a model upgrade fixing the wrong category of problem, because nobody stopped to classify what kind of failure they were actually looking at before reaching for a fix.

This is the feedback loop's actual job: not just noticing that something failed — the confidence flag from c12-04-2 already did that — but routing the failure to the right diagnosis before anyone spends a week on the wrong fix. A feedback loop that only alerts ("confidence dropped, someone should look") is half of a self-correction mechanism. The other half is a structure for figuring out why, quickly, instead of reaching for whatever fix happens to be most available (a model upgrade is always available; understanding a specific feed's bitrate quirk takes actual investigation, which is exactly why teams skip it under time pressure).

Five categories, and why the category determines the fix

When something fails, the fix that actually works depends on which of five categories the failure belongs to, and treating them interchangeably wastes real effort: context (the agent was missing information it needed, or had wrong information — fixed by writing, selecting, or isolating context better, per c12-02, not by touching the model); model (the agent had the right context but still made a poor judgment call — fixed by routing to a stronger model for that specific task, or improving the prompt's clarity); tool (a tool call failed, returned malformed data, or was misused — fixed by improving the tool's error handling or its description, not the model or the context); spec (the underlying requirement was ambiguous or actually wrong — fixed by revising the spec itself, the way c12-01 covered); environment (an external system was down, rate-limited, or behaving unexpectedly — fixed by retry logic or a fallback, not by touching the agent's reasoning at all). The bitrate example above was a context failure dressed up as a model problem, and the fix that would have actually worked — surfacing bitrate or a pre-transcription quality check into context — was never tried because the category was misdiagnosed from the start.

The diagnostic question before any fix
Before changing anything, ask: if I gave a human the exact same context, tools, and spec that the agent had, would they have made the same mistake? If yes, the problem is upstream of the model — context, tool, spec, or environment. If a human with identical inputs would clearly have done better, it's a model-category failure, and only then does 'use a stronger model' actually address the cause.

An agent repeatedly fails to complete a task because a tool call to a payment-validation API times out roughly 15% of the time under normal load. A team responds by rewriting the agent's system prompt to be more detailed. What's wrong with this response, and what would actually help?

A 15% timeout rate on an external API call is squarely an environment (or tool, depending on framing) failure — the agent's reasoning, context, and prompt were never the bottleneck. Rewriting the system prompt does nothing to fix an external service's reliability; the actual fix (retry logic, backoff, investigating the API itself) lives entirely outside the model and context layer this team focused on.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The outer-loop calibration pattern, Build It — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Agent