Phase 7: Responsible AI, Governance & Risk · 50 min · EU AI Act · NIST AI RMF · Python
The Problem
A PM at a mid-size insurtech company is scoping a feature: an LLM that drafts a recommended claim-approval decision, which a human adjuster can accept, edit, or override. Legal asks the obvious question — "is this high-risk under the EU AI Act?" — and the PM has two bad options: guess (risky, since fines for the most serious violations run up to the greater of €35 million or 7% of global annual turnover), or hand it to outside counsel and wait three weeks for an answer that a five-minute structured lookup should have produced in the design review.
Neither is necessary. The EU AI Act and the NIST AI Risk Management Framework are both, for a PM's purposes, mechanical enough to triage yourself for the first pass — not to replace legal sign-off on anything that actually matters, but to walk into that legal conversation already knowing the likely tier, the likely obligations, and exactly which specific fact (does a human really override this, or just rubber-stamp it?) will decide the answer. That's the skill research on AI PM job postings names explicitly: not compliance-officer depth, but the ability to turn "high-risk system" into a requirements list before legal gets involved.
The EU AI act: four tiers, decided by use case not technology
The EU AI Act does not regulate "AI" as a technology — it regulates specific use cases, and the tier is decided by what the system does to a person, not what's under the hood. A simple logistic regression screening job applicants is high-risk. A frontier LLM writing marketing copy is minimal-risk. The sophistication of the model is irrelevant to the tier.
| Tier | What triggers it | What it means for a PM |
|---|
| Unacceptable | Social scoring by public authorities, real-time biometric ID in public for law enforcement (narrow exceptions), manipulative/subliminal techniques causing harm, exploiting vulnerable groups | Cannot ship in the EU, full stop — this should be caught in the opportunity-assessment stage, not after a PRD is written |
| High | Annex III domains: biometric ID, critical infrastructure, education, employment/worker management, access to essential services (credit, insurance, benefits), law enforcement, migration, justice | Risk management system, data governance documentation, human oversight design, conformity assessment, logging, post-market monitoring — this becomes a real chunk of the roadmap, not a checkbox |
| Limited | Chatbots/conversational AI, synthetic/deepfake content, emotion recognition or biometric categorization | Disclosure obligations only: tell users they're talking to AI, label synthetic content |
| Minimal | Everything else | No mandatory obligations beyond existing law |
The claims-drafting example above is high-risk: insurance is an Annex III essential-service domain, and "the adjuster can technically override it" doesn't move the tier down if overrides are rare in practice — auditors specifically look for rubber-stamp oversight as a red flag that the human-in-the-loop requirement isn't real.
The five questions that get you 90% of the way there
(1) Does this fall into an Article 5 prohibited-practice category? (2) Does this operate in an Annex III domain — biometrics, employment, credit/insurance/benefits, education, law enforcement, critical infrastructure, migration, justice? (3) Does the system interact with users in a way that could be mistaken for a human, or generate synthetic media? (4) Is a human genuinely making the final call, or nominally reviewing an output they almost always accept? (5) Is the underlying model itself a general-purpose foundation model above the systemic-risk compute threshold — a separate obligation track from the application tier. Answer these five before you call legal, and the call becomes "confirm my triage" instead of "please figure this out."
NIST AI RMF: govern, map, measure, manage
Where the EU AI Act tells you what tier you're in, the NIST AI Risk Management Framework tells you how to actually manage the risk once you know the tier — and unlike the Act, it's voluntary guidance, not law, which makes it useful as a working process even for teams outside EU jurisdiction. It has four functions, and a PM has a real, specific job in each one:
- Govern — the org-level policies and accountability structure (who signs off on what, what's the escalation path). PM's job: know who the responsible-AI reviewer is before you need one, and make sure your feature's launch checklist references it.
- Map — understanding the context: what is this system for, who does it affect, what could go wrong. PM's job: this is your consequence-scan from Lesson 1, formalized — you're the one who actually knows the use case well enough to map it.
- Measure — tracking risk with actual metrics (this is where ethical KPIs from Lesson 1 and eval metrics from Phase 5 live). PM's job: make sure the metrics being measured are the ones that matter for this feature's actual risk profile, not a generic template.
- Manage — acting on what Measure found: mitigating, accepting, transferring, or avoiding the risk, with a documented decision. PM's job: this is the ship/mitigate/redesign call from Lesson 1 — Manage is where your decision becomes the org's decision of record.
The four functions aren't a one-time gate, they're a loop that runs for the life of the feature — Measure feeds back into Map as new failure modes surface post-launch, which is exactly the incident-response scenario in Lesson 4.
A team is scoping a feature that uses an LLM to summarize customer support tickets for internal triage — no customer ever sees the output, and a support lead manually assigns tickets after reading the summary. Under the EU AI Act's decision order, what should the PM check first, and what's the likely tier?
The Act's decision order is: prohibited practice check first, then Annex III domain check, then transparency triggers, and only minimal risk if none apply. Internal ticket summarization for triage isn't a prohibited practice and doesn't fall into an Annex III domain (it's not employment, credit, biometrics, law enforcement, etc.) — it's an operational efficiency tool, not a consequential decision about a person's access to services. It is not automatically a "chatbot" just because it uses an LLM; it never interacts with an end user at all, so the transparency-trigger path doesn't apply either. "Internal-only" isn't a blanket exemption in general — it just happens that this specific use case doesn't land in a regulated domain.
A high-risk-classified system has a documented human-oversight step, but the PM discovers the adjuster reviewing outputs accepts the AI's recommendation more than 95% of the time and spends under 10 seconds per review. Under the NIST AI RMF, which function's PM action does this fact most directly inform?
This is a Measure-function fact: NIST AI RMF's Measure function is about tracking risk with real metrics, and override/edit rate is precisely the metric this lesson calls out as necessary once human oversight exists. A 95%+ acceptance rate with under 10 seconds of review time is evidence the oversight may be nominal rather than real — exactly the kind of thing Measure is supposed to catch so it can feed into a Manage-function decision (mitigate, redesign, or accept with monitoring), not a reason to skip straight to shutdown without documenting the finding and its rationale.
Turning "high-risk" into a PRD
The number that actually matters to a PM isn't the tier name, it's the requirements list the tier produces. A high-risk classification under the Act, mapped through NIST's Measure/Manage functions, turns into concrete PRD line items: a documented human-oversight mechanism that's actually exercised (not rubber-stamped — track override rate as a real metric), a data governance section describing training/input data provenance, a logging requirement sufficient to reconstruct a decision after the fact, and a post-market monitoring plan with defined review cadence. None of that is exotic — it's the same shape of requirement a PM already writes for a launch checklist, just with a specific regulatory citation attached, which is exactly what makes it defensible to legal and fast to execute for engineering.
Build
Build it
The code artifact lesson_07_03_regulation_triage.py implements a first-pass EU AI Act and NIST AI RMF triage tool:
AISystemProfile — a structured description of a system: domain, prohibited-practice flags, transparency triggers, human-oversight reality (not just whether an override button exists)classify_eu_tier() — applies the Act's decision order to return a tier and the triggering ruleobligations_for_tier() — the concrete obligation checklist for that tiernist_function_actions() — for a given tier, the specific PM action under each of Govern/Map/Measure/Managerender_triage_report() — a markdown report suitable for a first legal conversation- A worked example: the insurance claims-drafting feature from the opening scenario
"""
lesson_07_03_regulation_triage.py
A first-pass EU AI Act risk-tier classifier plus NIST AI RMF
(Govern/Map/Measure/Manage) action mapper, built for a PM's initial
triage — not a replacement for legal sign-off on anything consequential.
References: EU AI Act (https://artificialintelligenceact.eu/),
NIST AI RMF (https://www.nist.gov/itl/ai-risk-management-framework).
Usage:
python lesson_07_03_regulation_triage.py
No external dependencies, no network calls, no API keys required.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
class EUTier(Enum):
UNACCEPTABLE = "Unacceptable"
HIGH = "High"
LIMITED = "Limited"
MINIMAL = "Minimal"
@dataclass
class AISystemProfile:
name: str
description: str
is_prohibited_practice: bool
annex_iii_domain: str | None # e.g. "employment", "credit", None if not applicable
has_transparency_trigger: bool # chatbot, synthetic media, emotion recognition
human_oversight_exists: bool
human_oversight_meaningfully_exercised: bool # rubber-stamp check
is_gpai_model_above_systemic_threshold: bool = False
def classify_eu_tier(profile: AISystemProfile) -> tuple[EUTier, str]:
"""Returns (tier, triggering_rule) following the Act's decision order."""
if profile.is_prohibited_practice:
return EUTier.UNACCEPTABLE, "Article 5 prohibited practice"
if profile.annex_iii_domain:
return EUTier.HIGH, f"Annex III domain: {profile.annex_iii_domain}"
if profile.has_transparency_trigger:
return EUTier.LIMITED, "Transparency-obligation use case (chatbot/synthetic media/emotion recognition)"
return EUTier.MINIMAL, "No prohibited practice, Annex III domain, or transparency trigger found"
def obligations_for_tier(tier: EUTier, profile: AISystemProfile) -> list[str]:
if tier == EUTier.UNACCEPTABLE:
return ["Cannot be placed on the market or used in the EU — redesign or cancel."]
if tier == EUTier.HIGH:
obligations = [
"Risk management system, maintained for the system's lifecycle",
"Data governance documentation (training/input data provenance)",
"Technical documentation sufficient to demonstrate compliance",
"Logging sufficient to reconstruct a decision after the fact",
"Human oversight mechanism that is genuinely exercised, not nominal",
"Conformity assessment and EU database registration before launch",
"Post-market monitoring plan with a defined review cadence",
]
if profile.human_oversight_exists and not profile.human_oversight_meaningfully_exercised:
obligations.append(
"FLAG: human oversight exists on paper but override/edit rate should be "
"measured — a rubber-stamp pattern will fail the human-oversight requirement"
)
return obligations
if tier == EUTier.LIMITED:
return [
"Disclose to users that they are interacting with an AI system",
"Label AI-generated or synthetic content as such",
]
return ["No mandatory obligations beyond existing law; voluntary codes of conduct encouraged."]
def gpai_obligations(profile: AISystemProfile) -> list[str]:
if not profile.is_gpai_model_above_systemic_threshold:
return []
return [
"Technical documentation for the general-purpose model",
"Training-data content summary",
"Copyright-compliance policy",
"Adversarial testing and systemic-risk mitigation (compute threshold triggered)",
"Incident reporting to the relevant authority",
]
def nist_function_actions(tier: EUTier) -> dict[str, str]:
"""The PM's specific action under each NIST AI RMF function, calibrated to tier."""
base = {
"Govern": "Confirm who the responsible-AI reviewer/sign-off owner is before launch; reference them in the launch checklist.",
"Map": "Run or update the consequence-scan (Lesson 1) for this specific use case and its affected groups.",
"Measure": "Define the ethical KPIs and, if human oversight exists, the override/edit rate as a tracked metric — not just accuracy.",
"Manage": "Document the ship/mitigate/redesign decision (Lesson 1's logic) as the org's decision of record, with an owner and a re-review date.",
}
if tier in (EUTier.HIGH, EUTier.UNACCEPTABLE):
base["Govern"] += " High-risk/unacceptable tiers should not ship on PM sign-off alone."
base["Measure"] += " High-risk tier requires this metric set in the PRD's launch criteria, not just a dashboard."
return base
def render_triage_report(profile: AISystemProfile) -> str:
tier, rule = classify_eu_tier(profile)
obligations = obligations_for_tier(tier, profile)
gpai = gpai_obligations(profile)
nist_actions = nist_function_actions(tier)
lines = [
f"# Regulation Triage — {profile.name}",
"",
profile.description,
"",
"## EU AI Act classification",
"",
f"**Tier: {tier.value}** — triggered by: {rule}",
"",
"### Obligations",
"",
]
lines += [f"- {o}" for o in obligations]
if gpai:
lines += ["", "### Separate track: GPAI systemic-risk obligations", ""]
lines += [f"- {o}" for o in gpai]
lines += ["", "## NIST AI RMF — PM actions by function", ""]
for fn, action in nist_actions.items():
lines.append(f"- **{fn}**: {action}")
lines += [
"",
"## Note",
"",
"This is a first-pass triage for scoping and PRD requirements, not a legal "
"determination. High/unacceptable-tier classifications should be confirmed "
"with legal before launch commitments are made.",
]
return "\n".join(lines)
# ── Worked example ───────────────────────────────────────────────────────────
# The insurance claims-drafting feature from the opening scenario: an LLM
# drafts a recommended claim-approval decision that an adjuster can accept,
# edit, or override — but rarely does in practice.
def build_example() -> AISystemProfile:
return AISystemProfile(
name="AI Claims Decision Drafting Assistant",
description=(
"An LLM drafts a recommended approve/deny decision and rationale "
"for insurance claims, which a human adjuster reviews before the "
"decision is finalized."
),
is_prohibited_practice=False,
annex_iii_domain="access to essential private services (insurance)",
has_transparency_trigger=True, # also drafts customer-facing explanation text
human_oversight_exists=True,
human_oversight_meaningfully_exercised=False, # adjusters accept as-is >90% of the time
is_gpai_model_above_systemic_threshold=False,
)
if __name__ == "__main__":
profile = build_example()
print(render_triage_report(profile))
print()
print(
"Takeaway: the Annex III domain check settles the tier as High before "
"the transparency trigger is even relevant — and the rubber-stamp flag "
"on human oversight is exactly the fact that should drive the PRD's "
"human-oversight requirement, not a generic 'human in the loop' line."
)