Phase 3: Data Strategy for AI Products · 40 min · Data audit framework · Python
Data Audits
Every AI feature idea is a data claim in disguise. Audit the claim before you roadmap the feature.
What you will learn
- Explain why an AI feature idea is implicitly a data claim, and why that claim needs to be tested before roadmapping
- Score a data source against a proposed AI feature on coverage, freshness, and labeling-readiness
- Distinguish a real data gap from a solvable one (buy, synthesize, or redesign the feature)
- Produce a data audit report that gives a go/no-go recommendation grounded in specific gaps
The Problem
A PM at a mid-size property management platform pitches a feature: "AI-powered maintenance request triage — the model reads a tenant's free-text description and routes it to the right vendor (plumber, electrician, HVAC) with a severity score." Leadership likes it. Design mocks look great. Engineering estimates six weeks. Everyone assumes the hard part is the model.
Three weeks in, the data science lead asks for the historical training set. There isn't one — the company's maintenance requests live in a legacy ticketing system as unstructured notes typed by property managers, not tenants, often summarizing a phone call two days after the fact. There's no "vendor category" field that's reliably filled in; half the records say "misc." There's no severity label at all — severity was always a judgment call the property manager made in their head. The feature that looked six weeks away is actually six weeks of model work sitting on top of an unknown amount of data-cleaning and labeling work that nobody scoped.
This is the single most common way AI features blow their timeline: the team scoped the model, not the data. A data audit is the PM discipline that prevents it — a structured pass over "what does this feature need" versus "what do we actually have," done before the feature gets a roadmap slot, not discovered by an engineer three weeks into the build.
What a data audit actually checks
A data audit isn't "do we have data" — almost every company has some data. It's whether the data that exists satisfies the specific requirements of the specific feature. Four dimensions do most of the work:
- Coverage — does the data span the cases the feature needs to handle? A support-ticket classifier trained only on tickets from your top three customer segments will fail silently on the other twelve.
- Freshness — is the data still representative of current reality? A pricing-recommendation feature trained on two-year-old transaction data may be modeling a market that no longer exists.
- Label quality — if the feature needs supervised signal (the maintenance triage example needs "correct vendor category" and "correct severity" per historical ticket), do those labels exist, and are they trustworthy? Free-text notes are not labels.
- Access and rights — is the team legally and technically allowed to use this data for this purpose? Data collected under one consent scope (e.g., "to service your account") can't automatically be repurposed to train a model (more in Lesson 4).
A feature can fail on any single dimension and the whole plan needs to change. Strong coverage with weak labels means a labeling program, not a training run. Strong labels with weak coverage means a data-collection sprint or a synthetic-data strategy before anything else happens.
The audit output is a decision, not a report
A data audit that just says "here's what we found" isn't finished. The deliverable is a recommendation: proceed as scoped, redesign the feature to fit the data that exists, delay and run a data-collection effort first, or kill the idea because the data gap isn't closeable in a reasonable timeframe. PMs who skip straight to "let's start labeling" without first asking "should this feature exist as scoped" waste the most expensive resource on the team — annotator and ML engineer time — chasing a feature that should have been redesigned in week one.
Reading a gap, not just a score
Numeric coverage scores are useful for triage but the real skill is diagnosing why a gap exists, because the fix depends on the cause. A severity field that's 90% "misc." might mean the field was never enforced in the UI (fixable — add a required dropdown going forward, backfill won't be perfect but new data will be clean) or it might mean severity genuinely isn't a stable concept for this domain (not fixable by better data collection — the feature needs to be redesigned around what's actually knowable).
The maintenance-triage team, after the audit, didn't kill the feature — they redesigned it. Instead of predicting severity from historical (unlabeled) data, they shipped a simpler v1: route by category only, using the ticket's stated issue type, which was reliably captured. Severity scoring became a v2 goal contingent on three months of clean, UI-enforced label collection. That's a data audit doing its job: not blocking the feature, but right-sizing it to what the data actually supports today.
A data audit finds a "vendor category" field that's 90% filled in as "misc." A PM has two competing theories: (1) the field was never enforced in the UI, so property managers skipped it, or (2) vendor category genuinely isn't a stable concept in this business — the same job often spans two vendor types. The PM doesn't investigate further and assumes theory 1, launching a UI fix requiring the field going forward. Three months later, the field is still 70% "misc." even on new tickets. What went wrong?
The lesson's core distinction is diagnosing why a gap exists before prescribing a fix, because the fix depends on the cause. A UI-enforcement fix only works if the cause is theory 1 (never enforced). If the real cause is theory 2 (the concept doesn't map cleanly onto the field as designed), forcing a required dropdown just forces property managers to pick an arbitrary wrong-ish answer instead of leaving it blank — the label quality doesn't actually improve, it just looks more confident. The audit should have tested both theories (e.g., a quick sample review of tickets to see if jobs genuinely span multiple categories) before committing to a fix, rather than assuming the cheaper-sounding explanation.
A PM's data audit for an AI resume-screening feature finds: (1) five years of historical hiring decisions with outcome labels (hired/not hired), (2) those decisions were made by human recruiters under the company's old, informal screening process, (3) the company was sued two years ago for discriminatory screening patterns in that same process. Coverage and label quality both score well. What should the audit recommend?
Coverage and label quality are necessary but not sufficient. A data audit that stops at "do we have enough labeled data" misses the fourth dimension — rights and appropriateness of use. Training a model to imitate decisions from a process already found legally discriminatory doesn't fix the bias, it automates and scales it. The right move isn't an automatic kill (the company might still build a carefully redesigned screening aid), but the audit must escalate this as a named risk requiring legal and responsible-AI review before engineering starts, not a footnote discovered after launch.
Build
Build it
The code artifact lesson_03_01_data_audits.py implements a data-audit scoring tool:
DataRequirement — one thing the feature needs from data: a dimension (coverage, freshness, label quality, rights), a description, and a weightDataSource — a candidate data source with per-requirement scores (0-100) and notes explaining the scoreFeatureAudit — bundles a feature name, its requirements, and the data sources being evaluated; computes a weighted readiness score and a recommendationrender_audit_report() — produces a markdown report: overall readiness score, per-dimension breakdown, flagged gaps, and a go/no-go/redesign recommendation- A worked example auditing a "smart maintenance request triage" feature for a property management platform, with deliberately mixed scores (strong category data, weak severity labels, a flagged rights gap on tenant-submitted photos)
"""
lesson_03_01_data_audits.py
A data-audit scoring tool for AI product features.
Models the PM discipline of checking what data a feature needs against
what data actually exists — before the feature gets a roadmap slot.
Produces a weighted readiness score per data source and a markdown
gap report with a go/no-go/redesign recommendation.
Usage:
python lesson_03_01_data_audits.py
No external dependencies, no network calls, no API keys required.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
class Dimension(Enum):
COVERAGE = "Coverage"
FRESHNESS = "Freshness"
LABEL_QUALITY = "Label Quality"
RIGHTS_ACCESS = "Rights & Access"
@dataclass
class DataRequirement:
dimension: Dimension
description: str
weight: float # relative importance, 0-1, should sum to 1.0 across requirements
@dataclass
class DimensionScore:
dimension: Dimension
score: float # 0-100
notes: str
@dataclass
class DataSource:
name: str
description: str
scores: dict[Dimension, DimensionScore] = field(default_factory=dict)
def add_score(self, dimension: Dimension, score: float, notes: str) -> None:
self.scores[dimension] = DimensionScore(dimension, score, notes)
@dataclass
class FeatureAudit:
feature_name: str
feature_description: str
requirements: list[DataRequirement]
sources: list[DataSource]
def weighted_readiness(self) -> float:
total = 0.0
for req in self.requirements:
source_score = 0.0
for source in self.sources:
ds = source.scores.get(req.dimension)
if ds:
source_score = max(source_score, ds.score)
total += source_score * req.weight
return total
def flagged_gaps(self, threshold: float = 60.0) -> list[tuple[Dimension, float, str]]:
gaps = []
for req in self.requirements:
best_score, best_notes = 0.0, "No source covers this dimension."
for source in self.sources:
ds = source.scores.get(req.dimension)
if ds and ds.score > best_score:
best_score, best_notes = ds.score, ds.notes
if best_score < threshold:
gaps.append((req.dimension, best_score, best_notes))
return gaps
def recommendation(self) -> str:
readiness = self.weighted_readiness()
rights_gap = any(
d == Dimension.RIGHTS_ACCESS and s < 60
for d, s, _ in self.flagged_gaps()
)
if rights_gap:
return "HOLD — rights/access gap must clear legal review before any modeling work starts"
if readiness >= 75:
return "PROCEED — data readiness supports the feature as scoped"
if readiness >= 45:
return "REDESIGN — right-size the feature to what the data actually supports today"
return "DELAY — run a dedicated data-collection or labeling effort before roadmapping this feature"
def render_audit_report(audit: FeatureAudit) -> str:
lines = [
f"# Data Audit — {audit.feature_name}",
"",
f"**Feature:** {audit.feature_description}",
"",
f"**Weighted readiness score:** {audit.weighted_readiness():.0f}/100",
f"**Recommendation:** {audit.recommendation()}",
"",
"## Requirements vs. best available source",
"",
"| Dimension | Weight | Best Score | Notes |",
"|---|---|---|---|",
]
for req in audit.requirements:
best_score, best_notes = 0.0, "No source covers this dimension."
for source in audit.sources:
ds = source.scores.get(req.dimension)
if ds and ds.score > best_score:
best_score, best_notes = ds.score, ds.notes
lines.append(
f"| {req.dimension.value} | {req.weight:.0%} | {best_score:.0f}/100 | {best_notes} |"
)
lines += ["", "## Data sources evaluated", ""]
for source in audit.sources:
lines.append(f"### {source.name}")
lines.append(source.description)
for dim, ds in source.scores.items():
lines.append(f"- **{dim.value}:** {ds.score:.0f}/100 — {ds.notes}")
lines.append("")
gaps = audit.flagged_gaps()
lines += ["## Flagged gaps (score < 60)", ""]
if not gaps:
lines.append("No dimension scored below threshold.")
else:
for dim, score, notes in gaps:
lines.append(f"- **{dim.value}** ({score:.0f}/100): {notes}")
return "\n".join(lines)
# ── Worked example ───────────────────────────────────────────────────────────
# "Smart Maintenance Triage" for a property management platform: route
# tenant maintenance requests to the right vendor category with a severity
# score. Deliberately mixed scores so the report has real substance.
def build_sample_audit() -> FeatureAudit:
requirements = [
DataRequirement(Dimension.COVERAGE, "Historical tickets spanning all vendor categories and property types", 0.3),
DataRequirement(Dimension.FRESHNESS, "Data reflects current vendor network and property mix", 0.15),
DataRequirement(Dimension.LABEL_QUALITY, "Reliable ground-truth vendor category and severity per ticket", 0.35),
DataRequirement(Dimension.RIGHTS_ACCESS, "Legal clearance to use tenant-submitted text/photos for training", 0.2),
]
legacy_tickets = DataSource(
"Legacy Ticketing System Export",
"5 years of maintenance tickets, free-text notes typed by property managers after phone calls.",
)
legacy_tickets.add_score(
Dimension.COVERAGE, 80,
"Spans all property types and most vendor categories; enterprise segment underrepresented.",
)
legacy_tickets.add_score(
Dimension.FRESHNESS, 70,
"Vendor network changed ~15% in the last year; older tickets reference retired vendors.",
)
legacy_tickets.add_score(
Dimension.LABEL_QUALITY, 35,
"Vendor category field is 'misc.' on ~40% of records; no severity field exists at all — "
"severity was an unrecorded judgment call.",
)
legacy_tickets.add_score(
Dimension.RIGHTS_ACCESS, 55,
"Tenant consent scope covers 'servicing your account'; using free-text descriptions to train "
"a model is a plausible secondary use but untested — flagged for legal review.",
)
tenant_photos = DataSource(
"Tenant-Submitted Photos (new mobile app feature)",
"Photos attached to maintenance requests submitted via the new tenant app, launched 4 months ago.",
)
tenant_photos.add_score(
Dimension.COVERAGE, 25,
"Only 4 months of data and only from tenants using the new app (~20% adoption so far).",
)
tenant_photos.add_score(
Dimension.FRESHNESS, 95,
"Fully current — reflects today's properties and vendor network.",
)
tenant_photos.add_score(
Dimension.LABEL_QUALITY, 20,
"No labels at all yet; would require a dedicated labeling pass.",
)
tenant_photos.add_score(
Dimension.RIGHTS_ACCESS, 40,
"App consent flow does not yet disclose AI training use for submitted photos — needs a "
"consent/ToS update before these can be used.",
)
return FeatureAudit(
feature_name="Smart Maintenance Request Triage",
feature_description=(
"Model reads a tenant's maintenance request and routes it to the correct vendor "
"category with a severity score, replacing manual property-manager triage."
),
requirements=requirements,
sources=[legacy_tickets, tenant_photos],
)
if __name__ == "__main__":
audit = build_sample_audit()
print(render_audit_report(audit))
print()
print(
"Takeaway: category routing is close to viable using the legacy ticket export once vendor "
"category is cleaned up, but severity scoring has no usable label source yet, and both "
"data sources have an open rights/access question that needs legal sign-off before any "
"modeling work starts. Recommendation: redesign v1 to category-only routing; treat severity "
"scoring and photo-based triage as v2, contingent on a labeling program and a consent update."
)
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy