Phase 9: Career, Portfolio & Interview Prep · 40 min · Evaluation vocabulary · Python
The Problem
"We've built a model that flags potentially fraudulent transactions. How would you evaluate whether it's ready to ship?" The candidate who answers "we'd run it on a test set and check the accuracy, and if accuracy is high enough, we ship" gets a polite nod and a mental red flag from the interviewer — not because the answer is wrong exactly, but because it's the answer of someone who hasn't internalized why accuracy alone is close to meaningless for a fraud-detection model, where the two error types (missing real fraud vs. flagging a legitimate transaction) have wildly different costs and a 99% "accurate" model that never flags anything can still be catastrophic if fraud is rare.
This is the specific interview round research names as one of four core tested signals for AI PM roles: not "can you write the evaluation code" (nobody expects that from a PM) but "can you hold a real, technically grounded conversation about evaluating a probabilistic system." The bar is fluency, not fluency in disguise as engineering competence — you need the vocabulary and judgment to ask the right questions and understand the answers, not to compute anything yourself.
Where the bar actually sits
A useful way to calibrate: an AI PM should be able to do everything a data scientist explaining their own eval results to them would expect a competent audience to do — follow the reasoning, catch an obviously wrong conclusion, and ask a pointed follow-up — without being able to independently reproduce the analysis. Concretely, that means being fluent (able to use correctly and explain simply, not necessarily compute by hand) in:
- Precision and recall, and specifically why they trade off against each other and which one matters more for a given feature's cost-of-error profile (fraud detection wants high recall even at some precision cost; a spam filter for an email client wants the opposite, since false-positiving a real email is worse than missing one spam message)
- Calibration — whether a model's stated confidence matches its actual accuracy (a model that says "90% confident" and is right 90% of the time is calibrated; one that says "90% confident" and is right 60% of the time is badly miscalibrated and dangerous to build UX around)
- Offline vs. online evaluation (Phase 5) — a golden test set catches known failure modes before launch; live A/B or shadow testing catches what the test set didn't anticipate, and a PM should know which question each answers
- The idea of a golden dataset as a maintained product asset, not a one-time artifact — and who owns keeping it current as the product and its failure modes evolve
"I'd ask the data science team for X" is a legitimate, strong answer
A common anxiety is that admitting "I don't know how to compute that myself, but I'd want to see it" reads as a weakness. It's the opposite — naming the specific thing you'd ask for (a confusion matrix broken out by transaction size, a precision-recall curve rather than a single threshold's numbers, a calibration plot) demonstrates you know exactly what evidence would change your decision, which is a stronger signal than vaguely gesturing at "more testing." The weak answer isn't "I'd ask an expert" — it's not knowing what to ask the expert for.
A worked mock answer
Interviewer: "We've built a model that flags potentially fraudulent transactions for manual review. How would you evaluate whether it's ready to ship?"
A strong PM answer, spoken naturally, hits several specific beats: "First I'd want to understand the cost asymmetry — missing real fraud versus flagging a legitimate transaction for review probably cost very different amounts, so I wouldn't evaluate this on accuracy alone. I'd want to see precision and recall separately, and I'd want to know where we're setting the decision threshold, since that's a lever we control that trades one against the other. Given this is fraud, I'd lean toward prioritizing recall — missing fraud is probably more expensive than an extra manual review — but I'd want the actual cost numbers from finance to confirm that instinct rather than assume it. I'd also want to know if the model's confidence score is calibrated, because if we're using a threshold on a confidence score that doesn't actually track real accuracy, our threshold tuning is built on sand. Beyond the offline metrics, I'd want a shadow-mode period before full launch — running the model against live traffic without acting on its output — to see if the offline numbers hold up on data the test set didn't anticipate. And I'd set a ship gate ahead of time: something like 'recall above X% at a false-positive rate the ops team can actually handle in review volume,' agreed with the team before we see the number, not negotiated after."
Notice what that answer does: uses correct vocabulary naturally (not as jargon-dropping), ties every technical concept back to a business tradeoff, asks for a specific number rather than a vague sanity check, and proposes a concrete ship gate — the eval-to-launch-decision skill from Phase 5's final lesson.
An interviewer says: "Our model has 96% accuracy predicting whether a support ticket needs escalation to a senior agent. What follow-up question would most efficiently reveal whether this model is actually usable?" Which question demonstrates the strongest technical fluency?
If escalations are rare (say, 5% of tickets), a model that never predicts escalation at all would already be 96% "accurate" while being completely useless — this is the class-imbalance trap accuracy hides, directly analogous to the fraud-detection framing in the worked answer. Asking for precision and recall specifically on the escalation class (not the aggregate) is the single most efficient follow-up to reveal whether the 96% figure means anything. The other options (language/framework, parameter count, raw training data) are either irrelevant to whether the model works or far too broad to be an efficient diagnostic question.
Red flags interviewers are listening for
A few patterns specifically signal under-preparation, independent of the specific question asked: treating "accuracy" as the only metric that exists; being unable to explain in plain language what precision and recall actually mean (not just naming them); proposing to "just add more data" as a universal fix without diagnosing what's actually wrong first; and confusing model-level metrics with business metrics (a model can have great precision/recall and still fail a business metric like revenue or retention if it's solving a mismeasured problem). None of these require deep ML expertise to avoid — they require having actually internalized Phase 5's material rather than memorizing term definitions the night before an interview.
A candidate says: "The model's confidence score says it's 90% sure this transaction is fraudulent, so I'd trust that and set our review threshold right at that number." What is missing from this reasoning, based on the calibration concept in this lesson?
A confidence score is only useful for threshold-setting if it's calibrated — meaning predictions labeled '90% confident' are actually right about 90% of the time. A model can be badly miscalibrated (e.g. everything it calls '90% confident' is only right 60% of the time) while still producing seemingly precise-looking numbers, and building a business-critical threshold on an unchecked, potentially miscalibrated score is exactly the trap this lesson's calibration point warns against. The fix isn't a specific alternate threshold — it's checking calibration before trusting the number at all.
Build
Build it
The code artifact lesson_09_03_technical_fluency_answer.py implements a technical-fluency answer builder and self-checker:
EvalQuestion — a technical interview prompt with the specific trap it's testing for (e.g. class imbalance, calibration blindness)FluencyAnswer — a candidate's structured answer: cost-asymmetry reasoning, specific metrics requested, a proposed ship gatecheck_fluency_answer() — flags the red-flag patterns this lesson calls outrender_answer_review() — a markdown self-review report- A worked example: the fraud-detection prompt, answered at the strong level shown above
"""
lesson_09_03_technical_fluency_answer.py
A technical-fluency interview answer builder and self-checker for AI PM
candidates. Structures a strong answer to "how would you evaluate this
model" around cost-asymmetry reasoning, specific requested metrics, and
a proposed ship gate, and checks a draft answer against the red-flag
patterns this lesson identifies.
References: Exponent (https://www.tryexponent.com/guides/microsoft-ai-product-manager-interview),
IGotAnOffer (https://igotanoffer.com/en/advice/ai-product-manager-interview).
Usage:
python lesson_09_03_technical_fluency_answer.py
No external dependencies, no network calls, no API keys required.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
class TrapType(Enum):
CLASS_IMBALANCE = "Class imbalance (accuracy hides a rare-class failure)"
CALIBRATION_BLINDNESS = "Confidence scores that don't track real accuracy"
OFFLINE_ONLY = "Relying only on a static test set with no live validation"
@dataclass
class EvalQuestion:
prompt: str
trap_type: TrapType
context_note: str
@dataclass
class FluencyAnswer:
question: EvalQuestion
names_cost_asymmetry: bool # discusses which error type matters more, and why
requests_specific_metrics: list[str] # e.g. ["precision", "recall", "calibration plot"]
proposes_shadow_or_online_test: bool
proposes_ship_gate: str # a concrete threshold agreed in advance, or "" if none
only_mentions_accuracy: bool = False
proposes_more_data_without_diagnosis: bool = False
def check_fluency_answer(answer: FluencyAnswer) -> tuple[int, list[str]]:
"""Scores the answer out of 5 and lists specific red flags found."""
score = 5
flags: list[str] = []
if answer.only_mentions_accuracy:
score -= 2
flags.append("RED FLAG: treats accuracy as the only relevant metric")
if not answer.names_cost_asymmetry:
score -= 1
flags.append("Never discussed which error type (false positive vs false negative) costs more and why")
if len(answer.requests_specific_metrics) < 2:
score -= 1
flags.append("Requested fewer than 2 specific metrics — too vague to demonstrate real fluency")
if not answer.proposes_shadow_or_online_test:
score -= 1
flags.append("Relied entirely on offline evaluation with no live/shadow validation step proposed")
if not answer.proposes_ship_gate:
score -= 1
flags.append("No concrete ship gate proposed — answer doesn't end in an actionable decision rule")
if answer.proposes_more_data_without_diagnosis:
score -= 1
flags.append("RED FLAG: proposed 'more data' as a fix without first diagnosing the actual failure mode")
return max(score, 0), flags
def render_answer_review(answer: FluencyAnswer) -> str:
score, flags = check_fluency_answer(answer)
lines = [
"# Technical Fluency Answer Review",
"",
f"**Question:** {answer.question.prompt}",
f"**Trap being tested:** {answer.question.trap_type.value}",
f"**Context:** {answer.question.context_note}",
"",
"## Answer structure",
"",
f"- Names cost asymmetry: {'Yes' if answer.names_cost_asymmetry else 'No'}",
f"- Specific metrics requested: {', '.join(answer.requests_specific_metrics) or 'None'}",
f"- Proposes shadow/online validation: {'Yes' if answer.proposes_shadow_or_online_test else 'No'}",
f"- Proposed ship gate: {answer.proposes_ship_gate or 'None stated'}",
"",
f"## Score: {score}/5",
"",
]
if flags:
lines.append("Flags to address:")
lines += [f"- {f}" for f in flags]
else:
lines.append("No red flags found — this answer demonstrates the expected AI PM technical fluency bar.")
return "\n".join(lines)
# ── Worked example ───────────────────────────────────────────────────────────
# The fraud-detection prompt, answered at the strong level shown in the lesson.
def build_example() -> FluencyAnswer:
question = EvalQuestion(
prompt="We've built a model that flags potentially fraudulent transactions for manual review. How would you evaluate whether it's ready to ship?",
trap_type=TrapType.CLASS_IMBALANCE,
context_note="Fraud is rare relative to legitimate transactions, so accuracy alone can hide a model that rarely flags anything.",
)
return FluencyAnswer(
question=question,
names_cost_asymmetry=True, # missing fraud vs. flagging a legitimate transaction
requests_specific_metrics=["precision", "recall", "calibration of confidence score"],
proposes_shadow_or_online_test=True,
proposes_ship_gate="Recall above X% at a false-positive rate the ops team can handle in review volume, agreed before seeing the number",
only_mentions_accuracy=False,
proposes_more_data_without_diagnosis=False,
)
if __name__ == "__main__":
answer = build_example()
print(render_answer_review(answer))
print()
print(
"Takeaway: a 5/5 answer here doesn't require computing anything — it "
"requires knowing that accuracy hides the fraud/legitimate cost "
"asymmetry, naming the two specific metrics that reveal it, proposing "
"a live validation step, and committing to a ship gate before seeing "
"the result. That's the technical fluency bar in practice."
)