Phase 7: Responsible AI, Governance & Risk · 45 min · Fairlearn · Consequence-scanning frameworks · Python

Bias, Fairness & Responsible AI Product Decisions

Ship metrics: accuracy went up. Ship reality: who got the false positive, and can they afford it?

Hiring signal: Responsible AI is now core PM scope, not a compliance sidebar — job postings for AI PM roles increasingly name consequence-scanning and fairness metrics as expected deliverables, not something legal hands back after the fact. PMs who can set an ethical KPI (equal opportunity gap, false-positive parity) with the same rigor as an accuracy target are the ones trusted to ship consequential features without a six-week legal review cycle.

What you will learn

Run a consequence-scan that identifies who is harmed when an AI feature fails, not just how often it fails
Set at least one ethical KPI (equal opportunity gap, false-positive parity) alongside a feature's accuracy target
Distinguish which fairness questions a PM can answer directly from which require commissioning a technical audit
Decide, using a documented threshold, whether a feature is ready to ship, needs mitigation, or needs to be redesigned

The Problem

A lending startup ships an AI underwriting assistant that speeds up loan pre-approval decisions. The model's overall accuracy is excellent — 94%, up from 89% with the old rules engine — and the PM presents it at the launch review as an unambiguous win. Three months later, a journalist finds that the model denies pre-approval to self-employed applicants at more than twice the rate of salaried applicants with comparable income and credit history, and the story runs under a headline that never mentions the word "accuracy" once.

Nothing in that story is a data science failure. The model does exactly what it was trained to do: minimize aggregate error. The failure is a product failure — nobody asked "when this model is wrong, who is it wrong about, and can they afford that?" before launch. That question, asked early enough to change the roadmap, is called consequence-scanning, and it is now a core AI PM deliverable, not something legal reviews after the PRD is final. The research is blunt about this: responsible AI has moved from compliance sidebar to core PM scope, sitting next to prioritization and roadmapping as a skill evaluators explicitly test for.

Consequence-Scanning: The One Question That Changes the Roadmap

Consequence-scanning is a structured version of one question, asked before launch instead of after a news story: if this feature is wrong, who gets hurt, how badly, and how would we know? It is not a vague ethics exercise — it produces a table you can put in a PRD.

For every feature with a decision that affects a person (approve/deny, rank/hide, flag/ignore, price up/price down), a consequence-scan asks:

Who is affected by a wrong output — not "users" in aggregate, but specific groups: self-employed applicants, non-native speakers, users on older devices, people in a specific age band, a specific protected class if the feature touches employment, housing, credit, or similar domains
What does "wrong" cost that person — a missed restaurant recommendation and a denied loan pre-approval are both "model errors," but one costs an afternoon and the other can cost a mortgage
Is the harm reversible — can the person appeal, retry, or get a human review, or is the decision final and consequential the moment it fires
Is the error distributed evenly — this is where fairness metrics enter: an even error rate across groups is very different from a model that's twice as wrong for one group

That fourth question is where "ethical KPIs" come from. Accuracy alone hides distribution. Two useful ones for a PM to know by name and set as real, tracked metrics:

Equal opportunity (or true positive rate parity): among people who should get a positive outcome (the loan should be approved, the resume should pass screening), does the model actually give it to them at similar rates across groups?
False-positive parity: among people who should get a negative outcome, is the model incorrectly flagging one group more than another? This is the one that shows up in the self-employed-applicant story above.

You don't need to compute these yourself — that's a data scientist's job, using tooling like Fairlearn, which implements exactly these metrics (demographic parity, equalized odds, equal opportunity) against a trained model. Your job as PM is to know these metrics exist, decide which ones matter for this feature's consequence profile, set a target threshold (e.g., "false-positive rate gap between any two groups must stay under 5 percentage points"), and put that threshold in the launch criteria next to the accuracy target — not as a nice-to-have, but as a ship-blocking metric with the same authority as latency or uptime.

Accuracy and fairness are not the same axis, and one does not imply the other
A model can hit 95% overall accuracy while being systematically worse for a specific group — in fact, this is the default outcome, not an edge case, whenever a group is underrepresented in training data or has a different underlying distribution of the target variable. Treating "accuracy is high" as sufficient evidence of "this is safe to ship" is the single most common responsible-AI mistake a PM makes, because it's an honest mistake — the dashboard genuinely does say the number is good. The fix isn't distrust of the dashboard; it's adding a second dashboard that slices the same metric by the groups your consequence-scan identified.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Deciding What a PM Owns vs What Needs an Audit, Setting Thresholds That Actually Block a Launch, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.