Phase 3: Data Strategy for AI Products · 40 min · Scale AI · Surge AI · Python

Labeling, Annotation & Human Feedback

Outsourcing the labeling doesn't outsource the accountability for label quality.

What you will learn

Compare labeling approaches (Scale AI, Surge AI, in-house) on cost, quality, turnaround, and domain fit
Design a quality-control process for labeled data, including gold sets and inter-annotator agreement
Explain RLHF at the level a PM needs: what preference data is, what a reward model does, and what decisions a PM actually makes in that pipeline
Produce a vendor comparison and QC scorecard that supports a labeling-program decision

The Problem

A fintech startup building an AI-powered transaction-categorization feature sends 50,000 transactions to a labeling vendor with a one-page instruction sheet: "categorize each transaction as one of: groceries, dining, transport, utilities, entertainment, other." Three weeks and $8,000 later, the labeled data comes back. The data science team starts training and the model plateaus at 71% accuracy — much worse than the 90%+ they expected from a supposedly simple categorization task.

The PM finally reads a sample of the labeled data closely. A $47 charge at "SHELL #4471" is labeled "transport" by one annotator and "other" by another. A $200 charge at "WALMART" is inconsistently split between "groceries" and "entertainment" depending on which annotator handled it — because Walmart sells both, and nobody defined a tie-breaking rule. The instructions never addressed ambiguous cases, there was no gold-standard set to calibrate annotators against, and nobody checked inter-annotator agreement before the whole batch shipped. The vendor did exactly what they were asked to do; the request itself was underspecified.

This is the failure mode that separates PMs who treat labeling as a procurement line-item from PMs who treat it as a specification-and-quality-control discipline. The vendor executes; the PM is accountable for whether what comes back is actually usable.

The vendor landscape

Three broad options exist, and most companies end up using more than one depending on task sensitivity and volume:

Managed labeling platforms like Scale AI — full-service annotation with a trained workforce, tooling, and QC built into their pipeline. Best for high-volume, well-specified tasks (bounding boxes, transcription, standard classification) where the marginal cost per label matters and the task doesn't require deep domain expertise.
Expert/specialized annotation platforms like Surge AI — positioned around higher-skill annotation and RLHF-style preference data, often used for tasks that need annotators who can genuinely evaluate response quality (writing, reasoning, code), not just apply a fixed taxonomy.
In-house annotation — your own team, domain experts, or contracted subject-matter reviewers. Slower and more expensive per label, but necessary when the task requires proprietary context (internal policy knowledge, a company's specific product catalog) that no external vendor can be trained on quickly, or when the data is too sensitive to leave the company's environment.

The choice isn't just cost-driven. A medical-symptom classifier needs clinical judgment no general-purpose labeling workforce has; a general "is this email spam" classifier doesn't. Matching task difficulty and sensitivity to the right vendor tier is itself a PM decision, and getting it wrong either overpays for expertise you don't need or underpays for expertise the task actually requires.

Write instructions like you're the only calibration the annotator gets
Annotators never see your product, your users, or your edge cases — they see your instructions and the examples you provide. Every ambiguous case that isn't addressed in writing becomes a coin flip, multiplied across thousands of labels. The fix isn't a longer instruction doc; it's a short doc plus 15-20 worked examples that specifically cover the ambiguous cases (the Shell gas station, the Walmart split-purpose purchase), each with the correct label and a one-line reason. Budget time to write this before sending a single item to a vendor — it's cheaper than re-labeling.

Quality control is the PM's job, not the vendor's

Every serious labeling program needs three QC mechanisms, regardless of which vendor tier is used:

A gold-standard set — a small batch (50-200 items) labeled by the PM or a trusted domain expert, used to calibrate annotators before the real batch starts and to spot-check quality afterward. If annotators score below an agreement threshold against the gold set, the instructions get revised before scale-up, not after.
Inter-annotator agreement (IAA) — when possible, send a subset of items to 2-3 annotators independently and measure agreement (e.g., Cohen's kappa for two annotators, Fleiss' kappa for more). Low agreement on a category means the category definition itself is broken, not that annotators are careless.
A sampling-based audit after delivery — even with a good gold set and decent IAA, spot-check a random sample of the delivered batch before it touches a training run. Vendors' internal QC catches some errors; it doesn't catch systematic misunderstandings of your domain.

RLHF (reinforcement learning from human feedback) is the same discipline applied to preference data instead of category labels. Annotators are shown two (or more) model outputs for the same prompt and asked which is better — that preference data trains a reward model, which then shapes the policy model's future outputs. A PM doesn't build the reward model, but a PM typically does own the rubric annotators use to judge "better" (helpfulness, safety, tone, factuality — and how those trade off against each other), the disagreement-resolution process when annotators split, and the sampling strategy for which prompts get sent for preference labeling in the first place. Get the rubric wrong — for example, rewarding confident-sounding answers over correct ones — and RLHF will systematically optimize the model toward the wrong behavior, at scale, without anyone noticing until it ships.

A team runs RLHF preference labeling with a rubric that instructs annotators to prefer "the more detailed and confident-sounding" response whenever both responses seem plausible. Six weeks after deploying the resulting model, users report it frequently states incorrect facts with high confidence instead of expressing uncertainty. What went wrong, and whose decision was the root cause?

The annotators did exactly what the rubric told them to do — that's the point. When "more detailed and confident-sounding" is the tiebreaker for ambiguous or genuinely uncertain cases, annotators systematically prefer responses that sound authoritative over responses that correctly hedge or say "I don't know." That preference data then trains a reward model that reinforces confident-sounding output regardless of accuracy, and the policy model optimizes toward exactly that behavior at scale. The fix isn't retraining annotators or changing the reward model's architecture — it's rewriting the rubric to explicitly reward calibrated honesty (including appropriate hedging) over confident-sounding detail, which is a PM-owned decision, not a data-science one.

A labeling vendor delivers 10,000 labeled customer-support tickets for a "top user intent" classification task. The vendor's own QC report shows 94% agreement with their internal reviewers. The PM ships the data straight to training without further review. Two weeks later, the model performs poorly on tickets mentioning a newly launched product line. What is the most likely root cause?

94% internal agreement only tells you annotators were consistent with each other and with the vendor's existing instructions — it says nothing about whether the instructions themselves cover the current reality of your product. If the taxonomy and gold set predate a new product line, annotators were consistently applying an outdated category scheme, and the vendor's QC had no way to detect that gap because it wasn't part of what they were checking. This is exactly why a PM's own post-delivery audit (checking a sample against current product reality, not just against the vendor's own gold set) is a separate, necessary step — vendor QC and PM QC catch different failure modes.

Build

Build it

The code artifact lesson_03_02_labeling_vendor_scorecard.py implements a labeling-vendor comparison and QC scorecard:

VendorProfile — a labeling option (managed platform, expert platform, in-house) with cost-per-label, typical turnaround, and a domain-expertise rating
TaskRequirements — the task being labeled: volume, ambiguity level, domain-expertise need, and sensitivity (whether data can leave the company's environment)
score_vendor_fit() — scores a vendor against task requirements and flags disqualifying mismatches (e.g., sensitive data sent to a vendor without the right data-handling terms)
QCPlan — a quality-control plan: gold-set size, target inter-annotator agreement threshold, and post-delivery audit sample size
render_vendor_scorecard() — produces a markdown report comparing vendors for a task and recommending a QC plan
A worked example comparing Scale AI, Surge AI, and an in-house review team for labeling preference data on a healthcare-adjacent chatbot's responses

"""
lesson_03_02_labeling_vendor_scorecard.py

A labeling-vendor comparison and quality-control scorecard tool.

Scores candidate labeling vendors/approaches against a task's actual
requirements (volume, ambiguity, domain expertise, data sensitivity),
and generates a recommended QC plan (gold set size, inter-annotator
agreement target, audit sample). This is the artifact a PM produces
before committing budget and data to a labeling vendor.

Usage:
    python lesson_03_02_labeling_vendor_scorecard.py

No external dependencies, no network calls, no API keys required.
"""

from __future__ import annotations

from dataclasses import dataclass
from enum import Enum

class VendorType(Enum):
    MANAGED_PLATFORM = "Managed Labeling Platform"
    EXPERT_PLATFORM = "Expert/RLHF Annotation Platform"
    IN_HOUSE = "In-House Review Team"

@dataclass
class VendorProfile:
    name: str
    vendor_type: VendorType
    cost_per_label_usd: float
    typical_turnaround_days: float
    domain_expertise_rating: int  # 1-5, 5 = handles deep domain expertise well
    handles_sensitive_data: bool  # has data-handling terms suitable for sensitive/regulated data

@dataclass
class TaskRequirements:
    task_name: str
    volume: int
    ambiguity_level: int  # 1-5, 5 = highly ambiguous, needs nuanced judgment
    domain_expertise_needed: int  # 1-5
    data_sensitivity: int  # 1-5, 5 = highly sensitive/regulated (e.g. health, financial)
    budget_usd: float
    deadline_days: float

@dataclass
class VendorScore:
    vendor: VendorProfile
    fit_score: float  # 0-100
    disqualified: bool
    disqualification_reason: str
    notes: list[str]

def score_vendor_fit(vendor: VendorProfile, task: TaskRequirements) -> VendorScore:
    notes: list[str] = []
    disqualified = False
    reason = ""

    # Hard disqualifiers
    if task.data_sensitivity >= 4 and not vendor.handles_sensitive_data:
        disqualified = True
        reason = (
            f"{vendor.name} lacks data-handling terms for sensitivity level "
            f"{task.data_sensitivity}/5 — cannot be used for this task regardless of cost."
        )

    total_cost = vendor.cost_per_label_usd * task.volume
    if total_cost > task.budget_usd:
        notes.append(
            f"Projected cost ${total_cost:,.0f} exceeds budget ${task.budget_usd:,.0f} "
            f"— would require volume reduction or budget increase."
        )

    if vendor.typical_turnaround_days > task.deadline_days:
        notes.append(
            f"Turnaround {vendor.typical_turnaround_days:.0f}d exceeds deadline "
            f"{task.deadline_days:.0f}d — would slip the timeline."
        )

    # Fit scoring: expertise match matters more as ambiguity/expertise need rises
    expertise_gap = max(0, task.domain_expertise_needed - vendor.domain_expertise_rating)
    expertise_penalty = expertise_gap * 15 * (task.ambiguity_level / 5)

    cost_efficiency = max(0, 100 - (vendor.cost_per_label_usd * 20))
    speed_score = max(0, 100 - (vendor.typical_turnaround_days * 5))

    fit_score = max(0, (cost_efficiency * 0.3 + speed_score * 0.2 + (100 - expertise_penalty) * 0.5))

    if expertise_gap > 0:
        notes.append(
            f"Domain expertise gap: task needs {task.domain_expertise_needed}/5, "
            f"vendor rated {vendor.domain_expertise_rating}/5 — expect lower label quality "
            f"without additional calibration and tighter QC."
        )

    return VendorScore(vendor, round(fit_score, 1), disqualified, reason, notes)

@dataclass
class QCPlan:
    gold_set_size: int
    iaa_target: float  # target inter-annotator agreement, e.g. 0.8 = Cohen's kappa 0.8
    audit_sample_size: int
    audit_sample_pct: float

    @staticmethod
    def recommend(task: TaskRequirements) -> "QCPlan":
        # Higher ambiguity -> bigger gold set and stricter agreement target
        gold_set = 50 + task.ambiguity_level * 30
        iaa_target = 0.6 + (5 - task.ambiguity_level) * 0.05
        audit_pct = 0.02 + (task.data_sensitivity / 5) * 0.03
        audit_size = max(100, int(task.volume * audit_pct))
        return QCPlan(gold_set, round(min(iaa_target, 0.9), 2), audit_size, round(audit_pct, 3))

def render_vendor_scorecard(task: TaskRequirements, vendors: list[VendorProfile]) -> str:
    scores = [score_vendor_fit(v, task) for v in vendors]
    scores.sort(key=lambda s: (s.disqualified, -s.fit_score))
    qc = QCPlan.recommend(task)

    lines = [
        f"# Labeling Vendor Scorecard — {task.task_name}",
        "",
        f"**Volume:** {task.volume:,} items | **Budget:** ${task.budget_usd:,.0f} | "
        f"**Deadline:** {task.deadline_days:.0f} days",
        f"**Ambiguity:** {task.ambiguity_level}/5 | **Domain expertise needed:** "
        f"{task.domain_expertise_needed}/5 | **Data sensitivity:** {task.data_sensitivity}/5",
        "",
        "## Vendor comparison",
        "",
        "| Vendor | Type | Fit Score | Status | Notes |",
        "|---|---|---|---|---|",
    ]
    for s in scores:
        status = "DISQUALIFIED" if s.disqualified else "eligible"
        note_text = s.disqualification_reason if s.disqualified else "; ".join(s.notes) or "No major flags."
        lines.append(
            f"| {s.vendor.name} | {s.vendor.vendor_type.value} | {s.fit_score if not s.disqualified else '—'} "
            f"| {status} | {note_text} |"
        )

    best = next((s for s in scores if not s.disqualified), None)
    lines += ["", "## Recommendation", ""]
    if best:
        lines.append(
            f"**{best.vendor.name}** ({best.vendor.vendor_type.value}) scores highest among eligible "
            f"vendors at {best.fit_score}/100."
        )
    else:
        lines.append("No vendor is eligible as scored — revisit data sensitivity handling or vendor list.")

    lines += [
        "",
        "## Recommended QC plan",
        "",
        f"- **Gold-standard set:** {qc.gold_set_size} items, labeled by the PM or a trusted domain expert "
        f"before scale-up",
        f"- **Inter-annotator agreement target:** {qc.iaa_target} (Cohen's/Fleiss' kappa) before accepting "
        f"a batch",
        f"- **Post-delivery audit sample:** {qc.audit_sample_size:,} items ({qc.audit_sample_pct:.1%} of "
        f"volume), checked against current product reality, not just the vendor's own gold set",
    ]
    return "\n".join(lines)

# ── Worked example ───────────────────────────────────────────────────────────
# Labeling preference data (RLHF-style pairwise comparisons) for a
# healthcare-adjacent chatbot's responses to patient medication questions.

def build_sample_scorecard() -> tuple[TaskRequirements, list[VendorProfile]]:
    task = TaskRequirements(
        task_name="Preference Labeling — Medication Q&A Chatbot Responses",
        volume=20000,
        ambiguity_level=4,
        domain_expertise_needed=4,
        data_sensitivity=5,
        budget_usd=60000,
        deadline_days=21,
    )
    vendors = [
        VendorProfile("Scale AI", VendorType.MANAGED_PLATFORM, 1.20, 10, 3, True),
        VendorProfile("Surge AI", VendorType.EXPERT_PLATFORM, 2.50, 14, 4, True),
        VendorProfile("In-House Clinical Reviewers", VendorType.IN_HOUSE, 6.00, 25, 5, True),
    ]
    return task, vendors

if __name__ == "__main__":
    task, vendors = build_sample_scorecard()
    print(render_vendor_scorecard(task, vendors))
    print()
    print(
        "Takeaway: for high-ambiguity, high-domain-expertise, high-sensitivity preference labeling, "
        "the cheapest managed platform is rarely the right call even if it scores well on cost "
        "efficiency — expertise gaps compound with ambiguity. Surge AI's expert-annotation fit and "
        "in-house clinical review both deserve serious consideration despite higher per-label cost; "
        "the scorecard makes that tradeoff explicit instead of defaulting to the lowest bid."
    )

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.

Key terms in this lesson

RLHF