Phase 3: Data Strategy for AI Products · 45 min · GDPR.eu · CCPA (California OAG) · Python

Privacy, Consent & Data Governance for AI

You can delete a row from a database. You cannot delete a fact from a trained model's weights.

What you will learn

Explain the GDPR basics an AI PM needs: lawful basis for processing, data subject rights, and why 'right to erasure' is complicated by model weights
Explain CCPA/CPRA basics: opt-out of sale/sharing, and sensitive personal information categories
Assess training data provenance and licensing risk for a proposed AI feature
Design opt-out/opt-in UX that satisfies legal requirements without silently killing the data flywheel

The Problem

A PM at a consumer productivity app ships a feature that fine-tunes a per-user writing-style model on each user's document history, to auto-complete in their own voice. Six months later, a user in the EU exercises their GDPR right to erasure and asks the company to delete their data. Engineering deletes the user's documents from the database within the required window — and closes the ticket. Legal reopens it two weeks later: the user's writing style is still baked into a fine-tuned model checkpoint that's been deployed to production and used to generate completions for other users' shared workspace documents. The database deletion was real and correct. The actual request — "remove my information from your systems" — was not satisfied, because a fine-tuned model isn't a database row you can delete; it's a set of weights that absorbed a statistical trace of the training data and doesn't offer a clean way to subtract one user's contribution back out.

This is the specific way AI makes data governance harder than a traditional PM's mental model expects. Traditional data governance assumes deletion is mechanical: find the row, delete the row, done. AI governance has to account for the fact that some data — anything used in training or fine-tuning — leaves a trace that ordinary deletion doesn't reach, which means the PM's job shifts from "can we delete this" to "should this data have gone into training in the first place, and if so, under what documented consent." Getting this right before a feature launches is dramatically cheaper than discovering the gap after a regulator or a user's lawyer asks about it.

GDPR basics an AI PM actually needs

GDPR applies to any company processing personal data of EU residents, regardless of where the company is based. The AI-relevant basics:

Lawful basis for processing — every use of personal data needs a legal basis (consent, legitimate interest, contract necessity, etc.). "We need it to train a better model" is not automatically a lawful basis; it typically needs to be consent or a carefully justified legitimate-interest assessment, especially for training data that will influence outputs shown to other users.
Data subject rights — access, rectification, erasure, portability, and the right to object to automated decision-making. The erasure right is the one that collides hardest with ML: deleting a row is straightforward, but if that row's content was already baked into a trained model, true erasure may require retraining without that data, which is expensive and slow — which is exactly why the consent decision at collection time matters so much more for AI features than for traditional ones.
Automated decision-making (Article 22) — GDPR gives data subjects the right not to be subject to a decision based solely on automated processing that has legal or similarly significant effects, without meaningful human involvement. Any AI feature that approves/denies/scores people (credit, hiring, insurance) needs a human-in-the-loop path, not just a model output, to stay compliant.

The consent question belongs at data-collection time, not at model-training time
By the time engineering is ready to train, the honest question — "did we get consent for this specific use, including training a model on it" — is usually much harder and more expensive to answer well than it would have been at the point the data was first collected. Bake a training-use consent flag into data collection from day one, even for data you're not yet planning to use for training. Retrofitting consent onto data already collected for a narrower purpose is a legal and product mess that regularly delays or kills AI features late in the roadmap.

CCPA/CPRA basics for AI PMs

California's CCPA (amended by CPRA) is narrower than GDPR in some ways and stricter in others. The AI-relevant basics: California residents have the right to know what personal information is collected and how it's used, the right to delete it, and critically, the right to opt out of the "sale or sharing" of their personal information — and CPRA's definition of "sharing" is broad enough that using a third-party AI vendor (a labeling vendor, a foundation-model API where prompts might be logged for the vendor's own training) can trigger sharing obligations depending on the vendor's data-use terms. CPRA also creates a category of "sensitive personal information" (precise geolocation, health data, and others) that gets stricter handling requirements. The practical PM takeaway: before sending user data to any third-party AI vendor, check whether that vendor's terms constitute a "sale or sharing" under CPRA, and make sure the opt-out mechanism actually propagates to that vendor relationship — an opt-out that stops your own use of the data but doesn't stop a vendor's use is not compliant.

A company sends user support-ticket text to a third-party foundation-model API to generate draft responses. The vendor's terms of service state prompts "may be used to improve our models" unless the customer opts out at the account level. The company has not configured the opt-out, and individual end users have no way to opt out of their specific tickets being used this way. What's the governance gap?

The vendor's default terms create a real governance gap: end users' ticket content (which frequently contains personal information) may be used to improve a third-party's models, and the people whose data is actually being used (the end users) have no opt-out mechanism at all — only the company, at the account level, does. If the company hasn't configured that account-level opt-out, it's implicitly accepting the vendor's default data-use terms on behalf of every end user without those users' knowledge or consent, which is exactly the kind of gap that becomes a CCPA/CPRA "sharing" and consent problem. The fix isn't avoiding third-party AI vendors altogether — it's an explicit PM-level decision to configure the opt-out (or negotiate different terms) and disclose the practice to end users, made before the feature ships, not discovered after.

Designing opt-out/opt-in UX without killing the flywheel

Privacy requirements and data-flywheel design are in genuine tension, not an artificial one: the flywheel from Lesson 3 wants maximum usage signal; consent law wants users to have a real, unburied choice about whether their data is used that way. The PMs who handle this well don't treat it as "compliance vs. product" — they design consent UX that's specific enough to be legally meaningful and clear enough that users who do opt in are opting into something they understand, which tends to produce a smaller but higher-trust, higher-quality signal pool. A vague, buried "I agree to the Terms of Service" checkbox technically satisfies almost nothing meaningfully, produces no real informed consent, and is exactly the pattern regulators have been increasingly willing to challenge. A specific toggle — "Allow my conversations to be used to improve this feature for other users" with a one-line explanation of what that means — is more legally defensible and, in practice, often converts a meaningful fraction of users who would have quietly distrusted a vaguer request.

A PM proposes replacing a specific "Allow my conversations to be used to improve this feature for other users" toggle with a single broader checkbox reading "I agree to the Privacy Policy and Terms of Service," arguing it will increase the opt-in rate and give the data flywheel more signal. What is the strongest objection to this change?

A bundled, vague consent checkbox may produce a higher raw opt-in number, but it does so by asking users to agree to something they likely haven't read or understood in the specific context of AI-training use — which is precisely the pattern that fails a genuine informed-consent standard and invites regulatory challenge. It also produces a lower-trust, lower-quality signal pool in practice, since a meaningful share of those opt-ins don't reflect real understanding. The specific toggle is the better design not just for compliance but because the resulting signal is more reliable.

Build

Build it

The code artifact lesson_03_04_privacy_governance_checklist.py implements a training-data privacy and governance risk checklist:

DataSource — a dataclass describing a proposed training/fine-tuning data source: what it is, whether consent was captured for AI training use specifically, whether it includes EU or California residents, and whether any third-party vendor is involved
GovernanceIssue — a flagged risk with severity and a specific recommended fix
assess_data_source() — runs a DataSource against a rule set covering GDPR lawful basis, erasure feasibility, CCPA/CPRA sharing, and sensitive-category data, returning a list of GovernanceIssues
render_governance_report() — produces a markdown risk report across multiple data sources for a feature, with a launch-readiness verdict
A worked example: the fine-tuned per-user writing-style feature from this lesson's opening, assessed both in its original (risky) form and in a redesigned form with explicit training-use consent

"""
lesson_03_04_privacy_governance_checklist.py

A training-data privacy and governance risk checklist for AI PMs.

Assesses proposed training/fine-tuning data sources against GDPR and
CCPA/CPRA-relevant risk factors -- lawful basis, erasure feasibility,
third-party sharing, and sensitive data categories -- and produces a
prioritized governance report. This does not replace legal review; it's
the artifact a PM uses to catch and flag risk *before* legal review,
so a launch doesn't get killed the week it was supposed to ship.

References:
  - GDPR.eu: https://gdpr.eu/
  - California CCPA (Office of the Attorney General): https://oag.ca.gov/privacy/ccpa

Usage:
    python lesson_03_04_privacy_governance_checklist.py

No external dependencies, no network calls, no API keys required.
"""

from __future__ import annotations

from dataclasses import dataclass, field
from enum import Enum

class Severity(Enum):
    BLOCKER = "blocker"
    HIGH = "high"
    MEDIUM = "medium"
    LOW = "low"

@dataclass
class DataSource:
    name: str
    description: str
    used_for_training: bool
    training_use_consent_captured: bool  # explicit consent for AI-training use specifically
    includes_eu_residents: bool
    includes_ca_residents: bool
    includes_sensitive_categories: bool  # health, precise geolocation, etc.
    third_party_vendor_involved: bool
    vendor_data_use_terms_reviewed: bool
    erasure_process_defined: bool  # is there a documented process for handling erasure requests

@dataclass
class GovernanceIssue:
    severity: Severity
    issue: str
    recommended_fix: str

def assess_data_source(source: DataSource) -> list[GovernanceIssue]:
    issues: list[GovernanceIssue] = []

    if source.used_for_training and not source.training_use_consent_captured:
        issues.append(GovernanceIssue(
            Severity.BLOCKER,
            f"'{source.name}' is used for training/fine-tuning but no explicit consent for "
            f"AI-training use was captured — general product consent does not cover this.",
            "Add a specific training-use consent flag at the point of data collection, or "
            "restrict this source to non-training uses until consent is captured.",
        ))

    if source.used_for_training and (source.includes_eu_residents or source.includes_ca_residents):
        if not source.erasure_process_defined:
            issues.append(GovernanceIssue(
                Severity.HIGH,
                f"'{source.name}' includes EU/CA residents and feeds training, but no erasure "
                f"process is defined for what happens to model weights when a user requests deletion.",
                "Define an erasure process before launch: retraining cadence that excludes "
                "deleted users' data, or a documented justification for why erasure applies "
                "only to raw storage and not model weights, reviewed with legal.",
            ))

    if source.includes_sensitive_categories:
        issues.append(GovernanceIssue(
            Severity.HIGH,
            f"'{source.name}' includes sensitive personal information categories (health, "
            f"precise geolocation, or similar) under CPRA, which requires stricter handling.",
            "Apply CPRA sensitive-data handling requirements: purpose limitation, and give "
            "users the right to limit use of sensitive personal information specifically.",
        ))

    if source.third_party_vendor_involved and not source.vendor_data_use_terms_reviewed:
        issues.append(GovernanceIssue(
            Severity.BLOCKER,
            f"'{source.name}' involves a third-party vendor whose data-use terms have not "
            f"been reviewed — default vendor terms often permit using submitted data to "
            f"improve the vendor's own models, which can constitute CCPA/CPRA 'sharing.'",
            "Review the vendor's data-use terms explicitly; configure any available opt-out "
            "of vendor-side training use, and disclose the practice to end users if it remains.",
        ))

    if source.used_for_training and source.includes_ca_residents and source.third_party_vendor_involved:
        issues.append(GovernanceIssue(
            Severity.MEDIUM,
            f"'{source.name}' data reaching a third-party vendor may trigger CCPA/CPRA "
            f"'sale or sharing' obligations, requiring an end-user-facing opt-out, not just "
            f"an account-level one.",
            "Confirm end users have a real opt-out path for this specific use, not only an "
            "account-level toggle configured by the company.",
        ))

    return issues

def render_governance_report(feature_name: str, sources: list[DataSource]) -> str:
    all_issues: list[tuple[DataSource, GovernanceIssue]] = []
    for source in sources:
        for issue in assess_data_source(source):
            all_issues.append((source, issue))

    severity_order = {Severity.BLOCKER: 0, Severity.HIGH: 1, Severity.MEDIUM: 2, Severity.LOW: 3}
    all_issues.sort(key=lambda pair: severity_order[pair[1].severity])

    blockers = [pair for pair in all_issues if pair[1].severity == Severity.BLOCKER]
    verdict = (
        f"NOT LAUNCH-READY — {len(blockers)} blocker(s) must be resolved before launch"
        if blockers
        else "No blockers found — review remaining issues with legal before final sign-off"
    )

    lines = [
        f"# Privacy & Data Governance Report — {feature_name}",
        "",
        f"**Data sources assessed:** {len(sources)}",
        f"**Total issues flagged:** {len(all_issues)}",
        f"**Launch readiness:** {verdict}",
        "",
        "## Issues by severity",
        "",
        "| Source | Severity | Issue | Recommended Fix |",
        "|---|---|---|---|",
    ]
    if not all_issues:
        lines.append("| — | — | No issues flagged | — |")
    else:
        for source, issue in all_issues:
            lines.append(
                f"| {source.name} | {issue.severity.value.upper()} | {issue.issue} | "
                f"{issue.recommended_fix} |"
            )
    return "\n".join(lines)

# ── Worked example ───────────────────────────────────────────────────────────
# The per-user fine-tuned writing-style feature from the lesson's opening,
# assessed in its risky original form and a redesigned, consent-first form.

def build_sample_assessment() -> tuple[str, list[DataSource]]:
    original = DataSource(
        name="User document history (original design)",
        description="User's documents used to fine-tune a per-user writing-style model.",
        used_for_training=True,
        training_use_consent_captured=False,
        includes_eu_residents=True,
        includes_ca_residents=True,
        includes_sensitive_categories=False,
        third_party_vendor_involved=False,
        vendor_data_use_terms_reviewed=False,
        erasure_process_defined=False,
    )
    redesigned = DataSource(
        name="User document history (redesigned, consent-first)",
        description="Same documents, now gated behind an explicit training-use consent toggle "
                     "with a documented retraining-on-erasure process.",
        used_for_training=True,
        training_use_consent_captured=True,
        includes_eu_residents=True,
        includes_ca_residents=True,
        includes_sensitive_categories=False,
        third_party_vendor_involved=False,
        vendor_data_use_terms_reviewed=False,
        erasure_process_defined=True,
    )
    return "Per-User Writing Style Fine-Tuning", [original, redesigned]

if __name__ == "__main__":
    feature_name, sources = build_sample_assessment()
    print(render_governance_report(feature_name, sources))
    print()
    print(
        "Takeaway: the same underlying data source goes from a launch-blocking governance "
        "risk to a clean assessment purely by adding explicit training-use consent and a "
        "defined erasure process at design time -- the data itself didn't need to change, "
        "the consent and process design around it did."
    )

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.