The Problem
GDPR Article 35 requires a Data Protection Impact Assessment (DPIA) for any processing "likely to result in a high risk to the rights and freedoms of natural persons." The EU AI Act Article 10 adds specific data governance requirements for high-risk AI systems. If you're deploying an AI system that processes personal data — and almost all do — you need a structured assessment that identifies privacy risks and documents mitigations.
This is not a checkbox exercise. A PIA is a threat model: what data flows through your system, what could go wrong, and what you're doing about it.
GDPR article 35 DPIA requirements
| Requirement | What It Means for AI |
|---|
| Systematic description of processing | What data does the model use? How is it collected, stored, processed? |
| Assessment of necessity and proportionality | Is collecting this data necessary for the model's purpose? Could you achieve the same with less data? |
| Assessment of risks to data subjects | What happens if the model leaks training data? Could membership inference reveal sensitive information? |
| Measures to address risks | Differential privacy, federated learning, data minimization, encryption, access controls |
| Residual risk assessment | After mitigations, what risk remains? Is it acceptable? |
EU AI act article 10 data governance
| Requirement | What It Means for AI |
|---|
| Training, validation, testing data sets are relevant, representative, free of errors | Data quality assessment (connects to Phase 2 bias detection) |
| Data governance practices | Data collection, labeling, cleaning, and quality assurance documentation |
| Data subject to data protection | GDPR compliance for all personal data in training/validation/test sets |
| Bias examination | Article 10(2)(f) — bias testing (connects to Phase 2 fairness metrics) |
Building the PIA tool
"""
privacy_impact_assessment.py
Structured privacy impact assessment for AI systems.
Evaluates against GDPR Art. 35 and EU AI Act Art. 10.
"""
import json
from datetime import datetime
PIA_FRAMEWORK = {
"gdpr_article_35": {
"name": "GDPR Article 35 — Data Protection Impact Assessment",
"requirements": [
{
"id": "gdpr_35_1",
"requirement": "Systematic description of processing operations and purposes",
"questions": [
"What personal data does the AI system process?",
"What is the purpose of processing?",
"How is data collected, stored, and retained?",
"Who are the data subjects?",
],
},
{
"id": "gdpr_35_2",
"requirement": "Assessment of necessity and proportionality",
"questions": [
"Is each data field necessary for the model's purpose?",
"Could the same outcome be achieved with less personal data?",
"Is the retention period proportionate to the purpose?",
],
},
{
"id": "gdpr_35_3",
"requirement": "Assessment of risks to rights and freedoms of data subjects",
"questions": [
"What is the membership inference risk (from Phase 4 Lesson 1)?",
"Could model outputs reveal sensitive attributes about individuals?",
"What happens if the model is accessed by unauthorized parties?",
"Could the model be used to discriminate against individuals?",
],
},
{
"id": "gdpr_35_4",
"requirement": "Measures to address identified risks",
"questions": [
"Is differential privacy applied to model training?",
"Is federated learning used to avoid centralizing data?",
"Are access controls and encryption in place?",
"Is data minimization applied (only necessary fields collected)?",
],
},
{
"id": "gdpr_35_5",
"requirement": "Residual risk assessment and sign-off",
"questions": [
"After mitigations, what is the residual privacy risk level?",
"Has the Data Protection Officer reviewed this assessment?",
"Is the residual risk acceptable for the intended use case?",
],
},
],
},
"eu_ai_act_article_10": {
"name": "EU AI Act Article 10 — Data and Data Governance",
"requirements": [
{
"id": "aia_10_1",
"requirement": "Training, validation, and testing data sets are relevant and representative",
"questions": [
"Has the training data been assessed for representativeness across protected groups?",
"Are there documented data quality checks?",
"Is the data relevant to the model's intended purpose?",
],
},
{
"id": "aia_10_2",
"requirement": "Data governance practices including design choices, data collection, labeling, cleaning, and quality assurance",
"questions": [
"Are data collection procedures documented?",
"Is data labeling/annotation documented with quality controls?",
"Are data cleaning and preprocessing steps documented?",
],
},
{
"id": "aia_10_3",
"requirement": "Data sets subject to data protection and privacy rights",
"questions": [
"Is all personal data in the training set processed in compliance with GDPR?",
"Has consent been obtained where required?",
"Are data subject rights (access, erasure, portability) supported?",
],
},
{
"id": "aia_10_4",
"requirement": "Bias examination per Article 10(2)(f)",
"questions": [
"Has a bias audit been conducted (Fairlearn/AIF360)?",
"Are fairness metrics documented in the model card?",
"Have bias mitigation measures been applied where needed?",
],
},
],
},
}
def conduct_pia(system_info, data_info, risk_assessment, mitigations):
"""
Conduct a structured privacy impact assessment.
Args:
system_info: Dict with system name, purpose, data subjects
data_info: Dict with data fields, collection, storage, retention
risk_assessment: Dict with identified risks and their severity
mitigations: Dict with applied mitigations and their effectiveness
Returns:
Dict with structured PIA results
"""
assessment = {
"system": system_info,
"data_processing": data_info,
"assessment_date": datetime.now().isoformat(),
"frameworks": {},
"risk_register": [],
"residual_risk": {},
"overall_recommendation": "",
}
# Evaluate each framework requirement
for framework_key, framework in PIA_FRAMEWORK.items():
req_results = []
for req in framework["requirements"]:
# Check if the requirement is addressed in the provided info
addressed = False
notes = ""
if "systematic description" in req["requirement"].lower():
addressed = bool(system_info.get("description") and data_info.get("fields"))
notes = f"System: {system_info.get('name', 'Unknown')}, Data fields: {data_info.get('fields', [])}"
elif "necessity" in req["requirement"].lower():
addressed = bool(data_info.get("necessity_justification"))
notes = data_info.get("necessity_justification", "Not documented")
elif "risks to" in req["requirement"].lower():
addressed = bool(risk_assessment.get("identified_risks"))
notes = f"{len(risk_assessment.get('identified_risks', []))} risks identified"
elif "measures" in req["requirement"].lower():
addressed = bool(mitigations.get("applied_measures"))
notes = f"{len(mitigations.get('applied_measures', []))} measures applied"
elif "residual" in req["requirement"].lower():
addressed = bool(risk_assessment.get("residual_risk_level"))
notes = f"Residual risk: {risk_assessment.get('residual_risk_level', 'Not assessed')}"
elif "representative" in req["requirement"].lower():
addressed = bool(data_info.get("representativeness_assessed"))
notes = data_info.get("representativeness_assessment", "Not assessed")
elif "governance" in req["requirement"].lower():
addressed = bool(data_info.get("governance_documented"))
notes = data_info.get("governance_docs", "Not documented")
elif "data protection" in req["requirement"].lower():
addressed = bool(data_info.get("gdpr_compliance"))
notes = data_info.get("gdpr_compliance_notes", "Not assessed")
elif "bias" in req["requirement"].lower():
addressed = bool(risk_assessment.get("bias_audit_conducted"))
notes = risk_assessment.get("bias_audit_reference", "Not conducted")
req_results.append({
"id": req["id"],
"requirement": req["requirement"],
"addressed": addressed,
"notes": notes,
"questions": req["questions"],
})
addressed_count = sum(1 for r in req_results if r["addressed"])
total = len(req_results)
assessment["frameworks"][framework_key] = {
"name": framework["name"],
"requirements": req_results,
"addressed": f"{addressed_count}/{total}",
"compliance_pct": round((addressed_count / total) * 100, 1),
}
# Build risk register
for risk in risk_assessment.get("identified_risks", []):
assessment["risk_register"].append({
"risk": risk.get("description"),
"severity": risk.get("severity"),
"likelihood": risk.get("likelihood"),
"mitigation": risk.get("mitigation"),
"residual_severity": risk.get("residual_severity"),
})
# Residual risk
assessment["residual_risk"] = {
"level": risk_assessment.get("residual_risk_level", "Not assessed"),
"justification": risk_assessment.get("residual_risk_justification", ""),
"dpo_reviewed": risk_assessment.get("dpo_reviewed", False),
}
# Overall recommendation
all_compliance = [f["compliance_pct"] for f in assessment["frameworks"].values()]
avg_compliance = sum(all_compliance) / len(all_compliance) if all_compliance else 0
if avg_compliance >= 80 and assessment["residual_risk"]["level"] in ["Low", "Very Low"]:
assessment["overall_recommendation"] = "APPROVED — Privacy controls are sufficient for deployment"
elif avg_compliance >= 60:
assessment["overall_recommendation"] = "CONDITIONAL — Address remaining gaps before deployment"
else:
assessment["overall_recommendation"] = "NOT APPROVED — Significant privacy gaps must be addressed"
return assessment
def generate_pia_report(assessment):
"""Generate a Markdown PIA report."""
lines = []
lines.append("# Privacy Impact Assessment Report\n")
lines.append(f"**System:** {assessment['system'].get('name', 'Unknown')}")
lines.append(f"**Purpose:** {assessment['system'].get('purpose', 'Not specified')}")
lines.append(f"**Date:** {assessment['assessment_date']}\n")
lines.append(f"## Overall Recommendation\n")
lines.append(f"**{assessment['overall_recommendation']}**\n")
# Framework compliance
for fkey, fdata in assessment["frameworks"].items():
lines.append(f"## {fdata['name']}\n")
lines.append(f"**Compliance:** {fdata['addressed']} ({fdata['compliance_pct']}%)\n")
lines.append("| Requirement | Addressed | Notes |")
lines.append("|---|---|---|")
for req in fdata["requirements"]:
status = "✅" if req["addressed"] else "❌"
lines.append(f"| {req['requirement']} | {status} | {req['notes']} |")
lines.append("")
# Risk register
if assessment["risk_register"]:
lines.append("## Risk Register\n")
lines.append("| Risk | Severity | Likelihood | Mitigation | Residual |")
lines.append("|---|---|---|---|---|")
for risk in assessment["risk_register"]:
lines.append(f"| {risk['risk']} | {risk['severity']} | {risk['likelihood']} | "
f"{risk['mitigation']} | {risk['residual_severity']} |")
lines.append("")
# Residual risk
lines.append("## Residual Risk Assessment\n")
lines.append(f"- **Level:** {assessment['residual_risk']['level']}")
lines.append(f"- **Justification:** {assessment['residual_risk']['justification']}")
lines.append(f"- **DPO Reviewed:** {'Yes' if assessment['residual_risk']['dpo_reviewed'] else 'No'}\n")
lines.append("---")
lines.append(f"*This assessment satisfies GDPR Article 35 and EU AI Act Article 10 documentation requirements.*")
return "\n".join(lines)
if __name__ == "__main__":
# Example: PIA for a loan approval model
system_info = {
"name": "LoanApproval-GBM-v1",
"purpose": "Automated loan approval for consumer credit",
"description": "Gradient boosting model that predicts loan repayment probability from applicant financial data",
"data_subjects": "Loan applicants (consumers)",
}
data_info = {
"fields": ["income", "credit_score", "debt_ratio", "employment_years", "repayment_history"],
"collection": "Internal loan application database",
"storage": "Encrypted database, access restricted to Risk Analytics team",
"retention": "7 years per financial regulations",
"necessity_justification": "All fields are required for credit risk assessment per banking regulations. Income and credit score are the primary predictors; debt_ratio and employment_years provide additional signal.",
"representativeness_assessed": True,
"representativeness_assessment": "Training data assessed for racial and gender representation in Phase 2 bias audit",
"governance_documented": True,
"governance_docs": "Data collection, cleaning, and quality assurance documented in data governance policy v2.1",
"gdpr_compliance": True,
"gdpr_compliance_notes": "Consent obtained at application time, data subject access rights supported via customer portal",
}
risk_assessment = {
"identified_risks": [
{
"description": "Membership inference attack could reveal that an individual applied for a loan",
"severity": "Medium",
"likelihood": "Medium",
"mitigation": "Model regularization applied (max_depth=5), differential privacy planned for v2",
"residual_severity": "Low",
},
{
"description": "Model outputs could reveal financial status to unauthorized parties",
"severity": "High",
"likelihood": "Low",
"mitigation": "Access controls, encryption, audit logging on all model endpoints",
"residual_severity": "Low",
},
{
"description": "Training data contains PII (income, credit score) that could be exposed in a breach",
"severity": "High",
"likelihood": "Low",
"mitigation": "Data encrypted at rest and in transit, access restricted to authorized personnel",
"residual_severity": "Medium",
},
],
"bias_audit_conducted": True,
"bias_audit_reference": "Phase 2 Fairlearn bias audit — DI ratio 0.72 (fails 80% rule), EO diff 0.04 (passes)",
"residual_risk_level": "Medium",
"residual_risk_justification": "After mitigations, residual risk is medium due to the disparate impact ratio failing the 80% rule and the PII exposure risk. Recommend deploying with bias mitigation (Phase 2 Lesson 5) and enhanced data encryption.",
"dpo_reviewed": True,
}
mitigations = {
"applied_measures": [
"Model regularization (max_depth=5) to reduce overfitting and membership inference risk",
"Data encryption at rest and in transit (AES-256)",
"Access controls and audit logging on model endpoints",
"Data minimization: only necessary fields collected",
"Bias audit conducted with Fairlearn (Phase 2)",
],
}
assessment = conduct_pia(system_info, data_info, risk_assessment, mitigations)
report = generate_pia_report(assessment)
with open("privacy_impact_assessment.md", "w") as f:
f.write(report)
with open("privacy_impact_assessment.json", "w") as f:
json.dump(assessment, f, indent=2)
print(f"Overall recommendation: {assessment['overall_recommendation']}")
for fkey, fdata in assessment["frameworks"].items():
print(f" {fdata['name']}: {fdata['addressed']} ({fdata['compliance_pct']}%)")
print(f" Risks identified: {len(assessment['risk_register'])}")
print(f" Residual risk: {assessment['residual_risk']['level']}")
print("\nReport saved to privacy_impact_assessment.md")
Your PIA tool reports 100% compliance on both GDPR Article 35 and EU AI Act Article 10 requirements, but the residual risk level is "Medium" and the DPO has not yet reviewed the assessment. Using the conduct_pia function's overall_recommendation logic, what recommendation would this produce, and is that the right call?
The conduct_pia function deliberately separates two different questions: "did we document everything the framework requires?" (compliance percentage) and "after our mitigations, how much risk is actually left?" (residual risk level). A system can score 100% on the documentation checklist and still carry real risk — for example, you can thoroughly document that your disparate impact ratio fails the 80% rule without that documentation making the underlying bias go away. That's exactly why the recommendation logic requires avg_compliance >= 80 AND residual_risk_level in ["Low", "Very Low"] before returning APPROVED — a Medium residual risk with full documentation correctly yields CONDITIONAL, meaning "keep working on the gaps before you deploy." A PIA that always says APPROVED whenever the paperwork is complete would be a rubber stamp, not a risk assessment.