Phase 6: Prompt Engineering & GenAI Feature Design · 50 min · Multi-step agents · Autonomy levels · Human-in-the-loop design
The Problem
A team ships an AI travel-booking agent: tell it your trip constraints, it searches flights, picks the best option, and books it — fully autonomously, because the demo where it "just handles everything end to end" tested extremely well with stakeholders. Two weeks after launch, it books a flight with a departure time that technically satisfies the user's stated constraint ("morning") but leaves at 5:40 AM, which is not what any reasonable person meant by morning, and the ticket is non-refundable. Nobody approved that specific booking before it happened. The team's response is "we'll add better prompting for what counts as morning" — a fix aimed at the wrong layer. The actual gap: the team never deliberately chose an autonomy level for an irreversible, financially consequential action. It defaulted to full autonomy because that's what the demo showcased, not because anyone decided a fully unattended booking flow was the right amount of oversight for this specific action.
Designing an agent product means making that choice deliberately, action by action, not once for the whole feature.
Multi-step agent architecture, at a product level
An agent, as distinct from a single prompt-response call, executes a loop: it plans a next step, acts (often via a tool call from Lesson 2), observes the result, and decides whether to continue, adjust the plan, or stop — repeating across multiple steps toward a goal, rather than producing one output for one input. The travel-booking agent's loop might look like: search flights -> evaluate options against constraints -> select one -> book it -> confirm to the user, each step depending on the result of the last.
The product risk this introduces that a single-call feature doesn't have: errors can compound across steps, and a wrong decision at step 2 (a training-data-adjacent misreading of "morning") can lead to an irreversible action at step 4 with no human ever having seen the full chain before it executed. This is the central product design problem of agents: as the number of autonomous steps between "user's stated goal" and "real-world consequence" grows, so does the value of an explicit checkpoint somewhere in that chain.
Autonomy levels: A framework PMs can actually use
Rather than a binary "autonomous or not," a useful framework treats autonomy as a spectrum, matched deliberately to each action's stakes and reversibility:
- Level 1 — Suggest only. The agent proposes an action but takes no action itself; a human executes it manually. Lowest risk, lowest leverage.
- Level 2 — Act, then report. The agent takes low-stakes, easily reversible actions autonomously (drafting an email, searching, summarizing) and simply reports what it did. Appropriate when a wrong action costs little to notice and undo.
- Level 3 — Act, with approval gate for consequential steps. The agent handles the full multi-step loop autonomously up to a defined checkpoint, then explicitly pauses for human approval before an irreversible or high-stakes action (the actual booking, the actual send, the actual charge), and continues automatically after approval.
- Level 4 — Fully autonomous. The agent completes the entire loop, including consequential and irreversible actions, with no human checkpoint. Appropriate only when every step is low-stakes and reversible, or when the cost of a delay (waiting for human approval) meaningfully outweighs the risk of an error.
The travel-booking agent's flaw was applying Level 4 to an action (a non-refundable booking) that squarely called for Level 3 — the search, filtering, and ranking steps are fine at high autonomy, but the actual purchase is exactly the kind of consequential, hard-to-reverse action that should pause for a human before executing.
Match autonomy to reversibility and stakes, not to what demos well
A fully autonomous agent is the most impressive demo and the most dangerous default, because a demo only ever shows the cases that go right. The right question for each individual action an agent can take isn't "can the model do this reliably" -- it's "if the model gets this one wrong, how bad is it, and can it be undone." A drafted-but-unsent email is trivially reversible; a non-refundable booking, a sent message to a customer, or a database write with no undo is not. High-autonomy behavior belongs on the reversible, low-stakes end of that spectrum; an approval gate belongs on the irreversible, high-stakes end -- regardless of how good the model has tested in practice.
An AI coding agent is being designed to autonomously find and fix bugs: it can read code, write a fix, run tests, and commit the fix to the main branch, all without human review, because "the tests will catch anything wrong." What autonomy level is this, and what is the strongest critique?
Committing directly to main with no human review is Level 4 autonomy applied to an action (a shipped code change) that isn't fully reversible in practice (once merged, it can affect other developers, trigger deployments, and require an explicit revert to undo -- a meaningfully different cost than never having merged it). Passing tests are a reasonable signal, but tests are well-known to be incomplete evidence of correctness, especially for subtle logic or edge-case regressions -- exactly the class of error human code review exists to catch. The fix isn't a better model; it's an approval gate (Level 3) at the commit step, letting the agent autonomously read code, propose a fix, and run tests, while a human still reviews before it merges.
Designing the escalation and approval UX
Once an approval gate exists, the UX of that gate matters as much as the decision to include one. A weak approval flow shows a wall of raw agent reasoning and asks a rushed human to approve or reject with no real ability to evaluate it, which trains users to rubber-stamp approvals without reading them — defeating the entire purpose of the gate. A strong approval flow surfaces the specific decision being made, the key facts behind it, and what would happen next, in a form a human can actually evaluate in the time they're likely to spend: "Book this flight: [airline] [flight number], departs 5:40 AM, $412, non-refundable. Approve / Choose a different option / Cancel search" is evaluable in five seconds. A raw log of the agent's multi-step reasoning is not.
Escalation design also needs an explicit answer for what happens when the agent is uncertain mid-task, not just at the final consequential step: does it pause and ask a clarifying question, take its best guess and flag low confidence, or stop entirely and hand off to a human? Each of these is a legitimate choice for different situations, but the product needs to have chosen one on purpose.
A customer support agent is designed to handle billing disputes autonomously: it can look up account details, calculate a refund amount, and issue the refund, all in one uninterrupted flow, with a summary shown to the user only after the refund is already issued. A support lead asks why approval happens after the action instead of before. The team says "showing it after is simpler to build and users like fast resolutions." What's the strongest response?
'Simpler to build' and 'users like fast resolutions' are real considerations, but they don't change the underlying stakes analysis: an issued refund is financially consequential and not trivially reversible (undoing an incorrect refund is a real operational cost, not a one-click action), which places it squarely in Level 3 territory -- autonomous handling up to the consequential step, then an approval gate before that step executes. Building the approval gate after the fact optimizes for engineering convenience and short-term UX at the expense of the exact protection the gate exists to provide. Doubling review isn't the fix either -- one well-placed pre-action gate at the right step is the design target, not redundant review at every step.