The Problem
In Lesson 1, you saw that models leak training data through membership inference. The fix is to make the model's output insensitive to any single individual's data — so that an attacker can't tell whether a specific person was in the training set. This is what differential privacy (DP) provides, with a mathematical guarantee.
The formal definition
A randomized algorithm $M$ is $\varepsilon$-differentially private if, for all datasets $D_1$ and $D_2$ that differ in at most one element (one person's data), and for all possible outputs $S$:
$$P(M(D_1) \in S) \leq e^{\varepsilon} \times P(M(D_2) \in S)$$
In plain English: the probability of any output is at most $e^{\varepsilon}$ times higher with your data than without it. When $\varepsilon$ is small (close to 0), $e^{\varepsilon} \approx 1 + \varepsilon$, meaning the output barely changes whether or not your data is included.
| Epsilon (ε) | Privacy Level | Typical Accuracy Impact | Use Case |
|---|
| 0.1 | Very strong | Significant (5-10% drop) | Medical research with strict privacy |
| 0.5 | Strong | Moderate (3-5% drop) | Census data, government statistics |
| 1.0 | Moderate | Small (1-3% drop) | Commercial ML with privacy requirements |
| 3.0 | Weak | Minimal (<1% drop) | Less sensitive applications |
| 10+ | Very weak | Negligible | Privacy in name only |
The laplace mechanism
The simplest way to achieve DP is the Laplace mechanism: add noise drawn from a Laplace distribution calibrated to the sensitivity of the query and the desired epsilon.
import numpy as np
def laplace_mechanism(true_value, sensitivity, epsilon):
"""
Add Laplace noise to achieve epsilon-differential privacy.
Args:
true_value: The true query result
sensitivity: How much the result can change with one person's data
epsilon: Privacy parameter (smaller = more private)
Returns:
Noisy value that satisfies epsilon-DP
"""
noise = np.random.laplace(0, sensitivity / epsilon)
return true_value + noise
Sensitivity is the maximum change in the query output when one person's data is added or removed. For a count query (e.g., "how many patients have diabetes?"), sensitivity = 1 (adding one person changes the count by at most 1). For a sum query, sensitivity = the maximum possible value.
The privacy budget
Each DP query "spends" some of your privacy budget. The composition theorem says:
- If you run $k$ queries each with $\varepsilon_i$-DP, the total privacy guarantee is $\sum \varepsilon_i$-DP
- This means you can't run unlimited queries — each one degrades the overall guarantee
- In practice, organizations set a total privacy budget (e.g., $\varepsilon_{total} = 1.0$) and allocate it across queries
Building the DP demo
"""
differential_privacy_demo.py
Demonstrates the privacy-utility tradeoff of differential privacy.
"""
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
def laplace_mechanism(true_value, sensitivity, epsilon):
"""Add Laplace noise calibrated to sensitivity and epsilon."""
noise = np.random.laplace(0, sensitivity / epsilon)
return true_value + noise
def dp_count(data, condition, epsilon):
"""
Differentially private count query.
Sensitivity = 1 (adding/removing one person changes count by 1).
"""
true_count = int(np.asarray(condition).sum()) # rows of `data` where the condition holds
return int(round(laplace_mechanism(true_count, 1, epsilon)))
def dp_mean(data, column, lower_bound, upper_bound, epsilon):
"""
Differentially private mean query.
Sensitivity = (upper_bound - lower_bound) / n (bounded range).
"""
n = len(data)
clipped = data[column].clip(lower_bound, upper_bound)
true_mean = clipped.mean()
sensitivity = (upper_bound - lower_bound) / n
return laplace_mechanism(true_mean, sensitivity, epsilon)
def demonstrate_privacy_utility_tradeoff():
"""Show how epsilon affects accuracy of DP queries."""
np.random.seed(42)
n = 10000
ages = np.random.normal(40, 15, n).clip(18, 80)
df = pd.DataFrame({"age": ages, "has_diabetes": np.random.binomial(1, 0.08, n)})
epsilons = [0.01, 0.1, 0.5, 1.0, 3.0, 10.0]
results = []
true_diabetes_count = df["has_diabetes"].sum()
true_mean_age = df["age"].mean()
for eps in epsilons:
# Run 100 trials and measure average error
count_errors = []
mean_errors = []
for _ in range(100):
dp_count_val = dp_count(df, df["has_diabetes"] == 1, eps)
count_errors.append(abs(dp_count_val - true_diabetes_count))
dp_mean_val = dp_mean(df, "age", 18, 80, eps)
mean_errors.append(abs(dp_mean_val - true_mean_age))
results.append({
"epsilon": eps,
"avg_count_error": round(np.mean(count_errors), 2),
"avg_mean_error": round(np.mean(mean_errors), 4),
"true_diabetes_count": int(true_diabetes_count),
"true_mean_age": round(true_mean_age, 2),
"privacy_level": "Very strong" if eps < 0.1 else "Strong" if eps < 0.5
else "Moderate" if eps < 1.0 else "Weak" if eps < 3.0 else "Very weak",
})
return results
def demonstrate_dp_training(epsilons=None):
"""Show how DP-SGD-style noise affects model accuracy."""
if epsilons is None:
epsilons = [0.1, 0.5, 1.0, 3.0, 10.0, 100.0]
np.random.seed(42)
n = 2000
X = np.random.randn(n, 10)
y = (X[:, 0] + X[:, 1] + np.random.randn(n) * 0.5 > 0).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
results = []
for eps in epsilons:
# Simulate DP-SGD by adding noise to training labels proportional to 1/epsilon
# (This is a simplified simulation — real DP-SGD clips gradients and adds noise to them)
noise_scale = 1.0 / max(eps, 0.01)
y_noisy = y_train.copy().astype(float)
# Add noise to features (simplified DP simulation)
X_noisy = X_train + np.random.laplace(0, noise_scale * 0.1, X_train.shape)
model = RandomForestClassifier(n_estimators=50, max_depth=5, random_state=42)
model.fit(X_noisy, y_noisy)
acc = model.score(X_test, y_test)
results.append({
"epsilon": eps,
"noise_scale": round(noise_scale, 4),
"test_accuracy": round(acc, 4),
"privacy_level": "Very strong" if eps < 0.1 else "Strong" if eps < 0.5
else "Moderate" if eps < 1.0 else "Weak" if eps < 3.0 else "Very weak",
})
return results
if __name__ == "__main__":
print("=== Differential Privacy: Privacy-Utility Tradeoff ===\n")
print("1. Query-level DP (count and mean queries):\n")
query_results = demonstrate_privacy_utility_tradeoff()
print(f"{'Epsilon':<10} {'Privacy':<15} {'Count Error':<15} {'Mean Error':<15}")
print("-" * 55)
for r in query_results:
print(f"{r['epsilon']:<10} {r['privacy_level']:<15} {r['avg_count_error']:<15} {r['avg_mean_error']:<15}")
print(f"\nTrue diabetes count: {query_results[0]['true_diabetes_count']}")
print(f"True mean age: {query_results[0]['true_mean_age']}")
print("\n2. Training-level DP (model accuracy with noise):\n")
train_results = demonstrate_dp_training()
print(f"{'Epsilon':<10} {'Privacy':<15} {'Noise Scale':<15} {'Test Accuracy':<15}")
print("-" * 55)
for r in train_results:
print(f"{r['epsilon']:<10} {r['privacy_level']:<15} {r['noise_scale']:<15} {r['test_accuracy']:<15}")
# Generate report
lines = ["# Differential Privacy Tradeoff Analysis\n"]
lines.append("## Query-Level DP\n")
lines.append("| Epsilon | Privacy Level | Avg Count Error | Avg Mean Error |")
lines.append("|---|---|---|---|")
for r in query_results:
lines.append(f"| {r['epsilon']} | {r['privacy_level']} | {r['avg_count_error']} | {r['avg_mean_error']} |")
lines.append(f"\nTrue diabetes count: {query_results[0]['true_diabetes_count']}")
lines.append(f"True mean age: {query_results[0]['true_mean_age']}\n")
lines.append("## Training-Level DP (Simplified)\n")
lines.append("| Epsilon | Privacy Level | Noise Scale | Test Accuracy |")
lines.append("|---|---|---|---|")
for r in train_results:
lines.append(f"| {r['epsilon']} | {r['privacy_level']} | {r['noise_scale']} | {r['test_accuracy']} |")
lines.append("\n## Key Findings")
lines.append("- As epsilon decreases (stronger privacy), query error increases")
lines.append("- At ε=0.01, the count error is very large — the noisy count is nearly useless")
lines.append("- At ε=1.0, the count error is moderate — the noisy count is useful for aggregate statistics")
lines.append("- At ε=10.0, the error is minimal — but the privacy guarantee is weak")
lines.append("- For model training, ε=1.0 provides a good balance: moderate privacy with minimal accuracy loss")
lines.append("\n## Recommendation")
lines.append("- For aggregate statistics (counts, means): ε = 0.5 to 1.0")
lines.append("- For ML model training: ε = 1.0 to 3.0 (use Opacus for real DP-SGD)")
lines.append("- For sensitive health data: ε = 0.1 to 0.5 (expect significant accuracy loss)")
with open("dp_tradeoff_report.md", "w") as f:
f.write("\n".join(lines))
print("\nReport saved to dp_tradeoff_report.md")
Your CISO asks you to set the epsilon for a differentially private model that will be trained on patient health records. The model will be used for treatment recommendations. What epsilon do you recommend, and why?
For health data, the privacy risk is high — membership inference could reveal that a specific person was a patient at a hospital, which is protected health information (PHI) under HIPAA. But the utility cost matters too — a treatment recommendation model that's 10% less accurate could harm patients. The right approach is to: (1) start with ε=1.0 as a reasonable default for sensitive data, (2) measure the actual accuracy impact on your model, (3) if the impact is acceptable (<3%), deploy with that epsilon, (4) if the impact is too high, try ε=2.0 or 3.0 and document the tradeoff, (5) present the measured tradeoff to the privacy officer and clinical leadership for a joint decision. Never pick epsilon in isolation — it's a risk decision that requires both privacy and clinical input.