Phase 3: Explainability & Transparency · 40 min · Python · Jinja2 · matplotlib

Building a Transparency Report Generator

The audit report is the artifact. The artifact is the evidence. The evidence is the compliance.

Real-world application: The ability to produce a professional, shareable HTML audit report that integrates fairness metrics, explanations, model cards, and risk assessments is the capstone deliverable for RAI engineering. This is what risk committees and auditors actually request.

What you will learn

Structure a transparency report for technical, executive, and regulatory audiences
Use Jinja2 to template an HTML report with embedded plots and metrics
Integrate fairness metrics, SHAP/LIME explanations, and model cards into one document
Produce a professional audit report as a shareable artifact

Objective

Test whether an explanation is faithful, understandable, decision-relevant, and appropriate for its actual audience.

The Problem

You now have a bias audit (JSON), SHAP explanations (plots + JSON), LIME explanations (plots + JSON), and a model card (Markdown + JSON). But a risk committee doesn't want to read 5 separate files. They want one professional report that integrates everything — the kind of document an auditor would request, a regulator would accept, and a CISO would file.

This is the transparency report — a single HTML document that combines all your audit artifacts into a shareable, professional deliverable.

Report structure for different audiences

Section	Technical Audience	Executive Audience	Regulatory Audience
Executive Summary	Brief	Detailed	Brief
Bias Audit Results	Full metrics + methodology	Pass/fail summary + key disparities	Full metrics + framework mapping
Explainability	SHAP plots + local explanations	Top 3 feature drivers	Article 13 compliance statement
Model Card	Full card	Key sections only	Annex IV mapping
Risk Assessment	Full risk register	Top 5 risks	FRIA/AIA summary
Recommendations	Technical mitigations	Business decisions needed	Compliance roadmap

Building the report generator with Jinja2

"""
transparency_report.py
Jinja2-templated HTML transparency report generator.
Integrates bias audit, SHAP/LIME, and model card into one document.
"""
import json
import base64
from datetime import datetime
from pathlib import Path

REPORT_TEMPLATE = """
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Responsible AI Audit Report — {{ model_name }}</title>
    <style>
        body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
               max-width: 900px; margin: 0 auto; padding: 40px; color: #1a1a1a; line-height: 1.6; }
        h1 { color: #1a1a2e; border-bottom: 3px solid #16213e; padding-bottom: 10px; }
        h2 { color: #16213e; margin-top: 40px; }
        h3 { color: #0f3460; }
        table { border-collapse: collapse; width: 100%; margin: 20px 0; }
        th, td { border: 1px solid #ddd; padding: 12px; text-align: left; }
        th { background-color: #16213e; color: white; }
        tr:nth-child(even) { background-color: #f8f9fa; }
        .pass { color: #28a745; font-weight: bold; }
        .fail { color: #dc3545; font-weight: bold; }
        .summary-box { background: #f0f4ff; border-left: 4px solid #16213e; padding: 20px; margin: 20px 0; }
        .plot { text-align: center; margin: 20px 0; }
        .plot img { max-width: 100%; border: 1px solid #ddd; border-radius: 4px; }
        .footer { margin-top: 60px; padding-top: 20px; border-top: 1px solid #ddd; color: #666; font-size: 0.9em; }
        .badge { display: inline-block; padding: 4px 12px; border-radius: 4px; font-size: 0.85em; font-weight: bold; }
        .badge-pass { background: #d4edda; color: #155724; }
        .badge-fail { background: #f8d7da; color: #721c24; }
    </style>
</head>
<body>
    <h1>Responsible AI Audit Report</h1>
    <p><strong>Model:</strong> {{ model_name }} | <strong>Date:</strong> {{ report_date }}</p>

    <div class="summary-box">
        <h2>Executive Summary</h2>
        <p><strong>Overall Assessment:</strong>
            <span class="badge {{ 'badge-pass' if overall_pass else 'badge-fail' }}">
                {{ 'PASS' if overall_pass else 'FAIL' }}
            </span>
        </p>
        <ul>
            <li><strong>Model accuracy:</strong> {{ accuracy }}</li>
            <li><strong>Disparate impact ratio:</strong> {{ di_ratio }}
                <span class="{{ 'pass' if di_pass else 'fail' }}">
                    ({{ 'passes' if di_pass else 'fails' }} 80% rule)
                </span>
            </li>
            <li><strong>Equalized odds difference:</strong> {{ eo_diff }}
                <span class="{{ 'pass' if eo_pass else 'fail' }}">
                    ({{ 'passes' if eo_pass else 'fails' }} 0.05 threshold)
                </span>
            </li>
        </ul>
    </div>

    <h2>1. Bias Audit Results</h2>
    <h3>Fairness Metrics by Protected Attribute</h3>
    <table>
        <tr><th>Attribute</th><th>Demographic Parity Diff</th><th>Equalized Odds Diff</th>
            <th>Disparate Impact Ratio</th><th>80% Rule</th><th>EO Threshold</th></tr>
        {% for attr, fm in fairness_metrics.items() %}
        <tr>
            <td>{{ attr }}</td>
            <td>{{ fm.demographic_parity_difference }}</td>
            <td>{{ fm.equalized_odds_difference }}</td>
            <td>{{ fm.disparate_impact_ratio }}</td>
            <td class="{{ 'pass' if fm.passes_80_percent_rule else 'fail' }}">
                {{ 'PASS' if fm.passes_80_percent_rule else 'FAIL' }}
            </td>
            <td class="{{ 'pass' if fm.passes_equalized_odds else 'fail' }}">
                {{ 'PASS' if fm.passes_equalized_odds else 'FAIL' }}
            </td>
        </tr>
        {% endfor %}
    </table>

    {% if bias_plot_b64 %}
    <div class="plot">
        <h3>Subgroup Performance Visualization</h3>
        <img src="data:image/png;base64,{{ bias_plot_b64 }}" alt="Bias Audit Plot">
    </div>
    {% endif %}

    <h2>2. Explainability — SHAP Analysis</h2>
    <h3>Global Feature Importance</h3>
    <table>
        <tr><th>Rank</th><th>Feature</th><th>Mean |SHAP|</th></tr>
        {% for feat, val in shap_importance %}
        <tr><td>{{ loop.index }}</td><td>{{ feat }}</td><td>{{ val }}</td></tr>
        {% endfor %}
    </table>

    {% if shap_plot_b64 %}
    <div class="plot">
        <h3>SHAP Global Summary Plot</h3>
        <img src="data:image/png;base64,{{ shap_plot_b64 }}" alt="SHAP Summary">
    </div>
    {% endif %}

    <h2>3. Model Card</h2>
    <div class="summary-box">
        <p><strong>Model:</strong> {{ model_card.model_details.name }}</p>
        <p><strong>Version:</strong> {{ model_card.model_details.version }}</p>
        <p><strong>Owner:</strong> {{ model_card.model_details.owner }}</p>
        <p><strong>Type:</strong> {{ model_card.model_details.model_type }}</p>
    </div>

    <h3>Intended Use</h3>
    <ul>
        {% for use in model_card.intended_use.primary_use_cases %}
        <li>{{ use }}</li>
        {% endfor %}
    </ul>

    <h3>Ethical Considerations</h3>
    <ul>
        {% for finding in model_card.ethical_considerations.bias_findings %}
        <li>{{ finding }}</li>
        {% endfor %}
    </ul>

    <h3>Caveats & Recommendations</h3>
    <h4>Known Limitations</h4>
    <ul>
        {% for lim in model_card.caveats_and_recommendations.known_limitations %}
        <li>{{ lim }}</li>
        {% endfor %}
    </ul>
    <h4>Monitoring Recommendations</h4>
    <ul>
        {% for mon in model_card.caveats_and_recommendations.monitoring_recommendations %}
        <li>{{ mon }}</li>
        {% endfor %}
    </ul>

    <h2>4. Framework Compliance Mapping</h2>
    <table>
        <tr><th>Framework</th><th>Control</th><th>Status</th></tr>
        <tr><td>NIST AI RMF</td><td>MEASURE 2.7 — Bias testing</td>
            <td class="pass">Evidence: This report</td></tr>
        <tr><td>EU AI Act</td><td>Art. 10(2)(f) — Bias examination</td>
            <td class="pass">Evidence: Section 1</td></tr>
        <tr><td>EU AI Act</td><td>Art. 13 — Transparency</td>
            <td class="pass">Evidence: Section 2 & 3</td></tr>
        <tr><td>EU AI Act</td><td>Annex IV — Technical documentation</td>
            <td class="pass">Evidence: Section 3 (Model Card)</td></tr>
        <tr><td>ISO 42001</td><td>Clause 9.1 — Monitoring & measurement</td>
            <td class="pass">Evidence: This report</td></tr>
    </table>

    <div class="footer">
        <p>Generated by rai-audit toolkit | {{ report_date }}</p>
        <p>This report satisfies NIST AI RMF MEASURE 2.7, EU AI Act Articles 10 & 13,
           and ISO 42001 Clause 9.1 evidence requirements.</p>
    </div>
</body>
</html>
"""

def encode_image_b64(filepath):
    """Encode an image file as base64 for embedding in HTML."""
    try:
        with open(filepath, "rb") as f:
            return base64.b64encode(f.read()).decode()
    except FileNotFoundError:
        return None

def generate_transparency_report(model_name, accuracy, fairness_metrics,
                                 shap_importance, model_card,
                                 bias_plot_path=None, shap_plot_path=None,
                                 output_file="audit_report.html"):
    """
    Generate a complete HTML transparency report.
    
    Args:
        model_name: Name of the model
        accuracy: Model accuracy (float)
        fairness_metrics: Dict from bias audit fairness_metrics
        shap_importance: List of (feature, value) tuples
        model_card: Dict from model card generator
        bias_plot_path: Path to bias audit plot PNG
        shap_plot_path: Path to SHAP summary plot PNG
        output_file: Output HTML filename
    
    Returns:
        Path to the generated report
    """
    from jinja2 import Template

    # Determine overall pass/fail
    all_pass = all(
        fm.get("passes_80_percent_rule") and fm.get("passes_equalized_odds")
        for fm in fairness_metrics.values()
    )

    # Get key metrics for summary
    first_attr = list(fairness_metrics.values())[0] if fairness_metrics else {}
    di_ratio = first_attr.get("disparate_impact_ratio", "N/A")
    eo_diff = first_attr.get("equalized_odds_difference", "N/A")
    di_pass = first_attr.get("passes_80_percent_rule", False)
    eo_pass = first_attr.get("passes_equalized_odds", False)

    # Encode images
    bias_plot_b64 = encode_image_b64(bias_plot_path) if bias_plot_path else None
    shap_plot_b64 = encode_image_b64(shap_plot_path) if shap_plot_path else None

    template = Template(REPORT_TEMPLATE)
    html = template.render(
        model_name=model_name,
        report_date=datetime.now().strftime("%Y-%m-%d %H:%M"),
        overall_pass=all_pass,
        accuracy=f"{accuracy:.2%}" if isinstance(accuracy, float) else accuracy,
        di_ratio=di_ratio,
        eo_diff=eo_diff,
        di_pass=di_pass,
        eo_pass=eo_pass,
        fairness_metrics=fairness_metrics,
        shap_importance=shap_importance[:10],
        model_card=model_card,
        bias_plot_b64=bias_plot_b64,
        shap_plot_b64=shap_plot_b64,
    )

    with open(output_file, "w") as f:
        f.write(html)

    return output_file

if __name__ == "__main__":
    # Example: Generate a report from saved artifacts
    try:
        with open("bias_audit_report.json") as f:
            bias_audit = json.load(f)
        with open("shap_explanations.json") as f:
            shap_results = json.load(f)
        with open("model_card.json") as f:
            model_card = json.load(f)
    except FileNotFoundError as e:
        print(f"Missing audit artifact: {e}")
        print("Run bias_audit.py, explain_shap.py, and model_card.py first.")
        exit(1)

    report_path = generate_transparency_report(
        model_name="LoanApproval-GBM-v1",
        accuracy=0.87,
        fairness_metrics=bias_audit.get("fairness_metrics", {}),
        shap_importance=shap_results.get("global_feature_importance_ranked", []),
        model_card=model_card,
        bias_plot_path="bias_audit_plot.png",
        shap_plot_path="shap_global_summary.png",
        output_file="audit_report.html",
    )
    print(f"Transparency report saved to {report_path}")

The report IS the compliance artifact
This single HTML report satisfies evidence requirements for NIST AI RMF MEASURE 2.7, EU AI Act Articles 10 & 13, and ISO 42001 Clause 9.1 simultaneously. This is the "implement once, evidence three times" pattern in action — one report, three frameworks. When an auditor asks "show me your bias testing evidence," you send them this file. When a regulator asks for Article 13 transparency documentation, you send them this file. When your risk committee asks for the model audit, you send them this file.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.