The Problem
An AI legal-research assistant confidently cites a case: "As established in Miller v. Thompson (2019), the statute of limitations for this claim is four years." A junior associate uses it in a client memo. The case doesn't exist — the model generated a plausible-sounding citation with no basis in its training data or any retrieval step, because nothing in the product ever told it that inventing a citation was worse than saying "I couldn't find a directly applicable precedent." The PRD for this feature had a full section on output formatting and tone. It had zero sentences about what the system should do when it doesn't actually know the answer. That omission wasn't a technical gap engineering forgot to fill — it was a product decision nobody made, so the model made the only decision it's statistically inclined to make by default: produce something confident and complete-shaped, because that's what its training rewards, absent an explicit instruction otherwise.
This is the core insight this lesson builds on: failure is not an edge case for a generative feature, it's a first-class product surface that needs the same design attention as the happy path. A deterministic feature fails loudly and obviously — a null pointer exception, a 500 error, a form that won't submit. A generative feature's most dangerous failure mode is the opposite: it fails quietly and confidently, producing something that looks exactly like a correct answer. Designing for failure means designing the specific mechanisms that make failure visible, contained, and escalatable instead of indistinguishable from success.
Hallucination UX patterns
Three patterns, usable individually or combined, make hallucination risk visible to the user instead of hidden inside confident prose:
- Citations/grounding — every factual claim links back to a specific source (a retrieved document, a specific line in a transcript, a database record). This doesn't prevent hallucination, but it makes it checkable: a user or reviewer can click through and verify, and a claim with no citation is itself a signal something's off.
- Confidence framing — language that reflects actual certainty instead of uniformly confident prose. "Based on the documents provided, it appears..." reads differently than a bare assertion, and that difference should track something real (retrieval quality, model-reported uncertainty where available) rather than being decorative hedging applied uniformly regardless of actual confidence.
- Uncertainty disclosure at the UI level — a visible indicator (not just careful wording) when the system is operating outside its comfortable zone: low retrieval match confidence, a question type the system historically gets wrong, or output that couldn't be verified against a grounding source.
Hedging language only works if it's calibrated — uniform hedging is worse than no hedging
If every single response says "I believe" or "it appears," those phrases stop carrying information and become filler users learn to ignore, exactly like a smoke detector that beeps constantly gets its battery pulled. The hedge has to be conditional — present when confidence is actually lower, absent when it's actually higher — which means the product needs some real signal driving it (retrieval match score, self-reported model confidence, whether a citation was found) rather than a stylistic instruction to "sound a bit uncertain sometimes." Design the confidence signal first; the UI language is just how that signal gets communicated.
Abstention: designing "I don't know"
Abstention is the specific product decision to have the system decline to answer, or answer only partially, when it doesn't have grounds for a confident response — and it needs to be an explicit, specified behavior, not an emergent hope. A well-specified abstention behavior names the trigger condition (no matching retrieval result above a similarity threshold; a question type outside the documented scope; conflicting information across sources) and the exact fallback response (a specific message, a suggestion to consult a human, a partial answer with the gap flagged). The counterintuitive part for PMs new to AI features: shipping a feature that abstains visibly and sometimes is usually a better user experience and a better business outcome than one that never visibly fails, because users calibrate their trust based on what they can observe — a system that's right 85% of the time and clearly flags the other 15% is more useful and more trusted than one that's right 90% of the time but gives no signal about which answers fall in the wrong 10%.
A PM is deciding whether to ship an AI diagnosis-support tool (for internal clinical staff, not patients) that answers with a confident recommendation on every query, versus a version that abstains and routes to a specialist when its retrieval confidence is below a threshold, reducing the percentage of queries it answers directly from 95% to 70%. Which version is the better product decision, and why?
In a context where a wrong confident answer can cause real harm, an abstention mechanism that visibly routes the hardest 25% of cases to a specialist is a stronger design than a system that answers everything with uniform confidence, even if the uniformly-confident system is technically 'right' more often in aggregate. The reason is trust calibration: users (here, clinical staff) need to know which answers to double-check, and a system that never signals uncertainty removes exactly the information they'd need to know when to be skeptical. The lower headline 'response rate' is the cost of a materially safer and more honest product.
Human-in-the-loop escalation tiers
Not every failure mode should escalate the same way — a well-designed escalation system has tiers keyed to specific, measurable triggers, not a single blanket "sometimes ask a human" rule. A workable structure has at least three tiers: auto-handle (the system acts autonomously — confidence is high, the action is low-risk or reversible), flag-for-review (the system produces an output but marks it for human review before it's used or shown externally — confidence is moderate, or the action has meaningful but recoverable consequences), and escalate-immediately (the system stops and hands off to a human before producing a user-facing output at all — confidence is low, or the action is high-risk/irreversible, such as anything touching money, legal exposure, or safety). Each tier needs a named, specific trigger (a confidence score threshold, a keyword/topic match, an action-type classification), not a vague "use judgment" instruction — because "use judgment" is exactly the kind of underspecified requirement that becomes an untested, undocumented default behavior the way the opening example's missing abstention spec did.
A PRD assigns a failure scenario to the "flag-for-review" tier with this fallback behavior: "the system should use good judgment about whether to show the output to the user." An engineer implements it as "always show the output, log a review flag in an internal dashboard nobody checks daily." Six weeks later, several flagged outputs with real errors reached users unreviewed. What's the root cause?
"Use good judgment" names a tier but not a mechanism -- it doesn't say whether the output should be blocked from the user until reviewed, shown with a visible pending-review indicator, or reviewed within some maximum time window before auto-releasing. The engineer's implementation (show immediately, log to an unmonitored dashboard) is a perfectly defensible literal reading of an underspecified instruction, which is exactly why vague fallback behaviors are flagged as a validation failure in a real failure-state spec. The fix is specifying the actual mechanism: e.g., "output is held from the user with a 'pending review' state until a reviewer approves or 4 hours elapse, whichever is first, and unreviewed-after-4-hours events page the on-call reviewer."
Error-state design beyond the model
Generative features also fail in mundane, non-hallucination ways that still need explicit design: the API call times out, the retrieval index is stale or unavailable, the output fails a guardrail check and gets blocked. Each of these needs a user-facing state that's honest about what happened without exposing internal implementation detail — "we couldn't generate a response right now, please try again" is better than a spinner that never resolves, and meaningfully better than silently returning a lower-quality fallback with no indication anything changed.
Build
Build it
The code artifact lesson_04_04_failure_state_designer.py implements a failure-state and escalation-tier design tool:
EscalationTier enum for AUTO_HANDLE, FLAG_FOR_REVIEW, and ESCALATE_IMMEDIATELYFailureScenario — a dataclass for one failure mode: trigger condition, the assigned escalation tier, and the specific fallback behaviorAbstentionRule — a dataclass for one abstention trigger and its exact fallback messageFailureDesignSpec — bundles failure scenarios and abstention rules for a feature, and validates every scenario has a tier-appropriate, specific (non-vague) fallbackrender_failure_design_spec() — produces a markdown failure-state design document- A worked example: the legal-research assistant from this lesson's opening, redesigned with citation requirements, an abstention rule for no-matching-precedent cases, and three-tier escalation
"""
lesson_04_04_failure_state_designer.py
A failure-state and escalation-tier design tool for AI PRDs.
Specifies failure modes as first-class product surfaces: named trigger
conditions, an assigned escalation tier (auto-handle / flag-for-review /
escalate-immediately), and a specific, non-vague fallback behavior for
each. Also specifies abstention rules -- when and how the system should
say "I don't know" rather than guess. This is the artifact that replaces
"the model should use good judgment" with a spec someone can actually
implement and test.
Reference: Anthropic, Reduce hallucinations —
https://docs.anthropic.com/en/docs/test-and-evaluate/strengthen-guardrails/reduce-hallucinations
Usage:
python lesson_04_04_failure_state_designer.py
No external dependencies, no network calls.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
class EscalationTier(Enum):
AUTO_HANDLE = "Auto-handle"
FLAG_FOR_REVIEW = "Flag for review"
ESCALATE_IMMEDIATELY = "Escalate immediately"
VAGUE_FALLBACK_MARKERS = ["use judgment", "use good judgment", "handle appropriately", "use discretion"]
@dataclass
class FailureScenario:
name: str
trigger_condition: str # specific, measurable trigger
tier: EscalationTier
fallback_behavior: str # the exact behavior/message, not a vague instruction
def is_specific(self) -> bool:
lower = self.fallback_behavior.lower()
return not any(marker in lower for marker in VAGUE_FALLBACK_MARKERS)
@dataclass
class AbstentionRule:
trigger_condition: str
fallback_message: str
@dataclass
class FailureDesignSpec:
feature_name: str
failure_scenarios: list[FailureScenario]
abstention_rules: list[AbstentionRule]
def vague_scenarios(self) -> list[FailureScenario]:
return [s for s in self.failure_scenarios if not s.is_specific()]
def tier_counts(self) -> dict[EscalationTier, int]:
counts = {t: 0 for t in EscalationTier}
for s in self.failure_scenarios:
counts[s.tier] += 1
return counts
def render_failure_design_spec(spec: FailureDesignSpec) -> str:
lines = [
f"# Failure-State Design Spec — {spec.feature_name}",
"",
"## Escalation tier summary",
"",
"| Tier | Scenario Count |",
"|---|---|",
]
counts = spec.tier_counts()
for tier in EscalationTier:
lines.append(f"| {tier.value} | {counts[tier]} |")
lines += ["", "## Failure scenarios", ""]
for s in spec.failure_scenarios:
specificity_flag = "" if s.is_specific() else " ⚠️ VAGUE FALLBACK — needs a specific behavior"
lines += [
f"**{s.name}** [{s.tier.value}]{specificity_flag}",
f"- Trigger: {s.trigger_condition}",
f"- Fallback behavior: {s.fallback_behavior}",
"",
]
lines += ["## Abstention rules", ""]
for r in spec.abstention_rules:
lines += [
f"- **Trigger:** {r.trigger_condition}",
f" - **Fallback message:** {r.fallback_message}",
]
vague = spec.vague_scenarios()
lines += ["", "## Validation", ""]
if not vague:
lines.append("All failure scenarios have specific, implementable fallback behaviors.")
else:
lines.append(f"**{len(vague)} scenario(s) have vague fallback behavior and need rework:**")
for s in vague:
lines.append(f"- {s.name}")
return "\n".join(lines)
# ── Worked example ───────────────────────────────────────────────────────────
# The legal-research assistant from the lesson's opening, redesigned with
# citation requirements, an abstention rule, and three-tier escalation.
def build_example_spec() -> FailureDesignSpec:
return FailureDesignSpec(
feature_name="Legal Research Assistant",
failure_scenarios=[
FailureScenario(
name="No matching precedent found",
trigger_condition="Retrieval returns no case with similarity score above 0.75 "
"for the legal question asked",
tier=EscalationTier.ESCALATE_IMMEDIATELY,
fallback_behavior="Response states: 'I could not find a directly applicable "
"precedent in the available case database. This question "
"should be researched manually or escalated to a senior "
"associate.' No case citation is generated.",
),
FailureScenario(
name="Moderate-confidence precedent match",
trigger_condition="Best retrieval match has similarity score between 0.75 and "
"0.90",
tier=EscalationTier.FLAG_FOR_REVIEW,
fallback_behavior="Response includes the citation with an explicit confidence "
"note: 'This is the closest match found (moderate confidence) "
"— verify against the primary source before citing in any "
"filing.' Flagged in the UI for supervising attorney review "
"before use.",
),
FailureScenario(
name="High-confidence precedent match",
trigger_condition="Best retrieval match has similarity score above 0.90 and "
"the case is confirmed still good law (not overturned)",
tier=EscalationTier.AUTO_HANDLE,
fallback_behavior="Response includes the citation directly with a hyperlink to "
"the full case text for verification.",
),
FailureScenario(
name="Retrieval index unavailable",
trigger_condition="Case database API call times out or returns an error",
tier=EscalationTier.ESCALATE_IMMEDIATELY,
fallback_behavior="UI shows: 'Case research is temporarily unavailable. Please "
"try again shortly or consult the research team directly.' "
"No fallback answer is generated from the model's parametric "
"knowledge alone.",
),
],
abstention_rules=[
AbstentionRule(
trigger_condition="No retrieval match above the 0.75 similarity threshold for "
"a case citation question",
fallback_message="'I could not find a directly applicable precedent in the "
"available case database.' — never generates a citation from "
"parametric knowledge alone.",
),
AbstentionRule(
trigger_condition="Question requires jurisdiction-specific knowledge outside "
"the documented scope (the tool is licensed for U.S. federal "
"case law only)",
fallback_message="'This question involves [jurisdiction], which is outside "
"this tool's documented scope (U.S. federal case law only). "
"Please consult a local-jurisdiction resource.'",
),
],
)
if __name__ == "__main__":
spec = build_example_spec()
print(render_failure_design_spec(spec))
print()
vague = spec.vague_scenarios()
print(
"Takeaway: every failure scenario has a specific, testable fallback tied to a "
"measurable trigger (a similarity-score threshold, an API error, a scope boundary) "
+ ("with no vague 'use judgment' fallbacks remaining." if not vague else
f"except: {[s.name for s in vague]}, which need rework before this spec ships.")
)