Phase 3: Data Strategy for AI Products · 40 min · Scale AI · Surge AI · Python

Labeling, Annotation & Human Feedback

Outsourcing the labeling doesn't outsource the accountability for label quality.

Hiring signal: AI PM interview loops explicitly test 'how you partner with data science across the model lifecycle' — and labeling/RLHF programs are where that partnership is most concrete, since the PM typically owns vendor selection, instructions, and QC while data science owns the model. Frontier lab postings (e.g., Anthropic Safeguards PM) call out the ability to 'get into the details on data' as a core bar; a PM who can design a QC sampling plan and defend a vendor tradeoff in an interview clears that bar directly.

What you will learn

Compare labeling approaches (Scale AI, Surge AI, in-house) on cost, quality, turnaround, and domain fit
Design a quality-control process for labeled data, including gold sets and inter-annotator agreement
Explain RLHF at the level a PM needs: what preference data is, what a reward model does, and what decisions a PM actually makes in that pipeline
Produce a vendor comparison and QC scorecard that supports a labeling-program decision

The Problem

A fintech startup building an AI-powered transaction-categorization feature sends 50,000 transactions to a labeling vendor with a one-page instruction sheet: "categorize each transaction as one of: groceries, dining, transport, utilities, entertainment, other." Three weeks and $8,000 later, the labeled data comes back. The data science team starts training and the model plateaus at 71% accuracy — much worse than the 90%+ they expected from a supposedly simple categorization task.

The PM finally reads a sample of the labeled data closely. A $47 charge at "SHELL #4471" is labeled "transport" by one annotator and "other" by another. A $200 charge at "WALMART" is inconsistently split between "groceries" and "entertainment" depending on which annotator handled it — because Walmart sells both, and nobody defined a tie-breaking rule. The instructions never addressed ambiguous cases, there was no gold-standard set to calibrate annotators against, and nobody checked inter-annotator agreement before the whole batch shipped. The vendor did exactly what they were asked to do; the request itself was underspecified.

This is the failure mode that separates PMs who treat labeling as a procurement line-item from PMs who treat it as a specification-and-quality-control discipline. The vendor executes; the PM is accountable for whether what comes back is actually usable.

The Vendor Landscape

Three broad options exist, and most companies end up using more than one depending on task sensitivity and volume:

Managed labeling platforms like Scale AI — full-service annotation with a trained workforce, tooling, and QC built into their pipeline. Best for high-volume, well-specified tasks (bounding boxes, transcription, standard classification) where the marginal cost per label matters and the task doesn't require deep domain expertise.
Expert/specialized annotation platforms like Surge AI — positioned around higher-skill annotation and RLHF-style preference data, often used for tasks that need annotators who can genuinely evaluate response quality (writing, reasoning, code), not just apply a fixed taxonomy.
In-house annotation — your own team, domain experts, or contracted subject-matter reviewers. Slower and more expensive per label, but necessary when the task requires proprietary context (internal policy knowledge, a company's specific product catalog) that no external vendor can be trained on quickly, or when the data is too sensitive to leave the company's environment.

The choice isn't just cost-driven. A medical-symptom classifier needs clinical judgment no general-purpose labeling workforce has; a general "is this email spam" classifier doesn't. Matching task difficulty and sensitivity to the right vendor tier is itself a PM decision, and getting it wrong either overpays for expertise you don't need or underpays for expertise the task actually requires.

Write instructions like you're the only calibration the annotator gets
Annotators never see your product, your users, or your edge cases — they see your instructions and the examples you provide. Every ambiguous case that isn't addressed in writing becomes a coin flip, multiplied across thousands of labels. The fix isn't a longer instruction doc; it's a short doc plus 15-20 worked examples that specifically cover the ambiguous cases (the Shell gas station, the Walmart split-purpose purchase), each with the correct label and a one-line reason. Budget time to write this before sending a single item to a vendor — it's cheaper than re-labeling.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Quality Control Is the PM's Job, Not the Vendor's, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

RLHF