Phase 7: Responsible AI, Governance & Risk · 45 min · Fairlearn · Consequence-scanning frameworks · Python

Bias, Fairness & Responsible AI Product Decisions

Ship metrics: accuracy went up. Ship reality: who got the false positive, and can they afford it?

What you will learn

Run a consequence-scan that identifies who is harmed when an AI feature fails, not just how often it fails
Set at least one ethical KPI (equal opportunity gap, false-positive parity) alongside a feature's accuracy target
Distinguish which fairness questions a PM can answer directly from which require commissioning a technical audit
Decide, using a documented threshold, whether a feature is ready to ship, needs mitigation, or needs to be redesigned

The Problem

A lending startup ships an AI underwriting assistant that speeds up loan pre-approval decisions. The model's overall accuracy is excellent — 94%, up from 89% with the old rules engine — and the PM presents it at the launch review as an unambiguous win. Three months later, a journalist finds that the model denies pre-approval to self-employed applicants at more than twice the rate of salaried applicants with comparable income and credit history, and the story runs under a headline that never mentions the word "accuracy" once.

Nothing in that story is a data science failure. The model does exactly what it was trained to do: minimize aggregate error. The failure is a product failure — nobody asked "when this model is wrong, who is it wrong about, and can they afford that?" before launch. That question, asked early enough to change the roadmap, is called consequence-scanning, and it is now a core AI PM deliverable, not something legal reviews after the PRD is final. The research is blunt about this: responsible AI has moved from compliance sidebar to core PM scope, sitting next to prioritization and roadmapping as a skill evaluators explicitly test for.

Consequence-scanning: the one question that changes the roadmap

Consequence-scanning is a structured version of one question, asked before launch instead of after a news story: if this feature is wrong, who gets hurt, how badly, and how would we know? It is not a vague ethics exercise — it produces a table you can put in a PRD.

For every feature with a decision that affects a person (approve/deny, rank/hide, flag/ignore, price up/price down), a consequence-scan asks:

Who is affected by a wrong output — not "users" in aggregate, but specific groups: self-employed applicants, non-native speakers, users on older devices, people in a specific age band, a specific protected class if the feature touches employment, housing, credit, or similar domains
What does "wrong" cost that person — a missed restaurant recommendation and a denied loan pre-approval are both "model errors," but one costs an afternoon and the other can cost a mortgage
Is the harm reversible — can the person appeal, retry, or get a human review, or is the decision final and consequential the moment it fires
Is the error distributed evenly — this is where fairness metrics enter: an even error rate across groups is very different from a model that's twice as wrong for one group

That fourth question is where "ethical KPIs" come from. Accuracy alone hides distribution. Two useful ones for a PM to know by name and set as real, tracked metrics:

Equal opportunity (or true positive rate parity): among people who should get a positive outcome (the loan should be approved, the resume should pass screening), does the model actually give it to them at similar rates across groups?
False-positive parity: among people who should get a negative outcome, is the model incorrectly flagging one group more than another? This is the one that shows up in the self-employed-applicant story above.

You don't need to compute these yourself — that's a data scientist's job, using tooling like Fairlearn, which implements exactly these metrics (demographic parity, equalized odds, equal opportunity) against a trained model. Your job as PM is to know these metrics exist, decide which ones matter for this feature's consequence profile, set a target threshold (e.g., "false-positive rate gap between any two groups must stay under 5 percentage points"), and put that threshold in the launch criteria next to the accuracy target — not as a nice-to-have, but as a ship-blocking metric with the same authority as latency or uptime.

Accuracy and fairness are not the same axis, and one does not imply the other
A model can hit 95% overall accuracy while being systematically worse for a specific group — in fact, this is the default outcome, not an edge case, whenever a group is underrepresented in training data or has a different underlying distribution of the target variable. Treating "accuracy is high" as sufficient evidence of "this is safe to ship" is the single most common responsible-AI mistake a PM makes, because it's an honest mistake — the dashboard genuinely does say the number is good. The fix isn't distrust of the dashboard; it's adding a second dashboard that slices the same metric by the groups your consequence-scan identified.

Deciding what a PM owns vs what needs an audit

Not every fairness question is a PM's to answer alone, and pretending otherwise is its own failure mode. A useful split:

PM-owned: identifying which groups a consequence-scan should check, setting the threshold for what's acceptable, deciding whether a gap found in testing blocks launch, and deciding what mitigation ships (more balanced training data, a fairness constraint in the model, a human review step for edge cases, or redesigning the feature so the AI recommends rather than decides)
Needs a technical audit: actually measuring the metric on real model outputs, diagnosing why a gap exists (data imbalance vs. proxy variables vs. genuine distributional differences), and validating that a proposed fix closes the gap without introducing a new one

A PM who tries to compute equalized odds by hand is wasting time; a PM who ships a consequential feature without ever asking whether one exists is the one who ends up in the news story.

An AI resume-screening tool has 91% accuracy at predicting which candidates a human recruiter would advance. During a consequence-scan, a PM learns the model was trained almost entirely on resumes from candidates with continuous employment histories, and the company's applicant pool includes many candidates with employment gaps (caregivers, veterans, career-changers). What is the PM's highest-leverage next step?

The consequence-scan surfaced a specific, plausible mechanism for disparate impact — a training set that underrepresents a group the tool will actually screen in production. The correct next step is to get evidence before deciding, not to guess: commission the specific fairness measurement (false-positive parity for the employment-gap subgroup) that would confirm or rule out the concern. Shipping on aggregate accuracy alone ignores exactly the failure mode this lesson describes. A disclaimer doesn't fix a systematic error, it just discloses one — and disclosure is a Lesson 2 topic, not a substitute for fixing a knowable gap here.

A fairness audit finds the loan pre-approval model has a false-positive parity gap of 9 points for self-employed applicants (above the 5-point target) and the affected harm is HIGH severity with only MODERATE reversibility (an appeal process exists but adds weeks of delay). The ML lead proposes rebalancing the training data and remeasuring before the next launch window. Is this the right call, or does it need to go further?

The decision logic in this lesson is explicit about this: SEVERE or HIGH severity combined with a hard-to-reverse (or here, only moderately reversible) outcome for an affected group crosses the line from "patch the model" to "redesign the feature." A data rebalancing effort is a reasonable MITIGATE-tier response for lower-severity or easily-reversible harms, but here it risks shipping the same class of harm again if the rebalancing doesn't fully close the gap — and by the time that's discovered, real applicants have already been wrongly denied with only a slow appeal path. The right move is a design change: e.g., routing self-employed applicants to mandatory human underwriter review rather than auto-denial, until the KPI passes with a comfortable margin.

Setting thresholds that actually block a launch

An ethical KPI that isn't a ship-blocking gate is just a slide. To make it real, a PM needs three things in the PRD: the metric, the threshold, and who signs off if it's missed. For example: "False-positive rate gap between any demographic group and the overall population must stay under 5 points at launch, remeasured monthly; a breach triggers automatic escalation to the responsible-AI review lead, not a silent retrain." That sentence does more work than a page of ethics-policy language, because it's checkable.

Build

Build it

The code artifact lesson_07_01_consequence_scan.py implements a consequence-scanning and ethical-KPI decision tool:

AffectedGroup — a group identified in the scan, with harm severity and reversibility
ConsequenceScan — the structured worksheet: feature description, affected groups, and per-group harm assessment
EthicalKPI — a fairness metric with a name, a measured value per group, and a target threshold
ship_decision() — combines the scan and KPI results into a ship / mitigate / redesign recommendation with rationale
render_consequence_report() — a markdown report a launch review would actually read
A worked example: an AI-assisted loan pre-approval feature, with a deliberately-found false-positive-parity gap

"""
lesson_07_01_consequence_scan.py

A consequence-scanning and ethical-KPI decision tool for AI product
features. Structures the "if this goes wrong, who gets hurt?" question
into a worksheet, tracks fairness metrics (e.g. false-positive parity,
equal opportunity) against a threshold, and produces a ship/mitigate/
redesign recommendation with a written rationale for a launch review.

References: ProdPad (https://www.prodpad.com/blog/ethics-in-ai/),
Fairlearn (https://fairlearn.org/).

Usage:
    python lesson_07_01_consequence_scan.py

No external dependencies, no network calls, no API keys required.
"""

from __future__ import annotations

from dataclasses import dataclass, field
from enum import Enum

class Severity(Enum):
    LOW = "Low (inconvenience, easily reversible)"
    MEDIUM = "Medium (real cost, reversible with effort)"
    HIGH = "High (major cost, hard to reverse)"
    SEVERE = "Severe (life-affecting, irreversible)"

class Reversibility(Enum):
    EASY = "Easy — retry, appeal, or human override readily available"
    MODERATE = "Moderate — appeal exists but has friction or delay"
    HARD = "Hard — no practical appeal path"

@dataclass
class AffectedGroup:
    name: str
    how_affected: str
    severity: Severity
    reversibility: Reversibility

@dataclass
class ConsequenceScan:
    feature_name: str
    feature_description: str
    affected_groups: list[AffectedGroup] = field(default_factory=list)

    def highest_severity(self) -> Severity:
        order = [Severity.SEVERE, Severity.HIGH, Severity.MEDIUM, Severity.LOW]
        present = {g.severity for g in self.affected_groups}
        for s in order:
            if s in present:
                return s
        return Severity.LOW

@dataclass
class EthicalKPI:
    name: str
    description: str
    group_values: dict[str, float]  # group name -> measured rate (0-1)
    max_allowed_gap: float  # e.g. 0.05 for a 5-point gap

    def observed_gap(self) -> float:
        values = list(self.group_values.values())
        return max(values) - min(values) if values else 0.0

    def passes(self) -> bool:
        return self.observed_gap() <= self.max_allowed_gap

    def worst_pair(self) -> tuple[str, str]:
        items = sorted(self.group_values.items(), key=lambda kv: kv[1])
        return items[0][0], items[-1][0]

def ship_decision(scan: ConsequenceScan, kpis: list[EthicalKPI]) -> tuple[str, str]:
    """Returns (decision, rationale). Decision is one of SHIP / MITIGATE / REDESIGN."""
    failing = [k for k in kpis if not k.passes()]
    severity = scan.highest_severity()

    if not failing:
        return "SHIP", (
            "All ethical KPIs are within their target gap, and the highest "
            f"identified harm severity is {severity.value}. No blocking "
            "fairness gap found; proceed with standard monitoring."
        )

    hard_to_reverse = any(
        g.reversibility == Reversibility.HARD for g in scan.affected_groups
    )
    if severity in (Severity.SEVERE, Severity.HIGH) and hard_to_reverse:
        names = ", ".join(k.name for k in failing)
        return "REDESIGN", (
            f"KPI(s) failing target: {names}. Harm severity is "
            f"{severity.value} with a hard-to-reverse outcome for at least "
            "one affected group. A mitigation patch is not sufficient at "
            "this severity/reversibility combination — the feature needs "
            "a design change (e.g. human-in-the-loop before consequential "
            "action, not just a rebalanced model)."
        )

    names = ", ".join(k.name for k in failing)
    return "MITIGATE", (
        f"KPI(s) failing target: {names}. Harm severity is "
        f"{severity.value} and at least one affected group has a "
        "reversible path. Do not ship as-is; ship after a specific "
        "mitigation (rebalanced training data, fairness constraint, or "
        "an added human-review step for the affected group) closes the "
        "gap, with the KPI remeasured before launch."
    )

def render_consequence_report(scan: ConsequenceScan, kpis: list[EthicalKPI]) -> str:
    decision, rationale = ship_decision(scan, kpis)
    lines = [
        f"# Consequence Scan — {scan.feature_name}",
        "",
        scan.feature_description,
        "",
        "## Affected groups",
        "",
        "| Group | How affected | Severity | Reversibility |",
        "|---|---|---|---|",
    ]
    for g in scan.affected_groups:
        lines.append(f"| {g.name} | {g.how_affected} | {g.severity.value} | {g.reversibility.value} |")

    lines += ["", "## Ethical KPIs", "", "| KPI | Groups | Observed gap | Max allowed | Pass? |", "|---|---|---|---|---|"]
    for k in kpis:
        group_str = ", ".join(f"{g}={v:.0%}" for g, v in k.group_values.items())
        status = "PASS" if k.passes() else "FAIL"
        lines.append(f"| {k.name} | {group_str} | {k.observed_gap():.1%} | {k.max_allowed_gap:.0%} | {status} |")

    lines += ["", f"## Decision: {decision}", "", rationale]
    return "\n".join(lines)

# ── Worked example ───────────────────────────────────────────────────────────
# An AI-assisted loan pre-approval feature. Deliberately includes a failing
# false-positive-parity KPI so the report has real substance to react to.

def build_example() -> tuple[ConsequenceScan, list[EthicalKPI]]:
    scan = ConsequenceScan(
        feature_name="AI Loan Pre-Approval Assistant",
        feature_description=(
            "Model scores loan applications and auto-denies pre-approval "
            "below a confidence threshold; approved applications route to "
            "a human underwriter for final sign-off."
        ),
        affected_groups=[
            AffectedGroup(
                name="Self-employed applicants",
                how_affected="Irregular income documentation reads as risk signal to the model",
                severity=Severity.HIGH,
                reversibility=Reversibility.MODERATE,
            ),
            AffectedGroup(
                name="First-time applicants (thin credit file)",
                how_affected="Sparse credit history scored similarly to poor credit history",
                severity=Severity.MEDIUM,
                reversibility=Reversibility.EASY,
            ),
        ],
    )
    kpis = [
        EthicalKPI(
            name="False-positive parity (wrongly denied pre-approval)",
            description="Among applicants who would ultimately be approved by a human underwriter, share wrongly auto-denied by the model.",
            group_values={"Salaried applicants": 0.04, "Self-employed applicants": 0.13},
            max_allowed_gap=0.05,
        ),
        EthicalKPI(
            name="Equal opportunity (correctly approved among qualified)",
            description="Among applicants who meet underwriting criteria, share correctly advanced to human review.",
            group_values={"Salaried applicants": 0.91, "First-time applicants": 0.87},
            max_allowed_gap=0.05,
        ),
    ]
    return scan, kpis

if __name__ == "__main__":
    scan, kpis = build_example()
    print(render_consequence_report(scan, kpis))
    print()
    print(
        "Takeaway: the false-positive parity gap for self-employed "
        "applicants (9 points, versus a 5-point target) is exactly the "
        "kind of finding that never shows up in an aggregate accuracy "
        "number, and exactly the kind of finding a consequence-scan is "
        "designed to catch before launch, not after a news story."
    )

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.