Phase 6: AI Red Teaming Methodology & Tools · 60 min · Python · PyRIT · MITRE ATLAS
The Problem
A traditional penetration test has a clean success criterion: did you get unauthorized access, exfiltrate data, or execute code you shouldn't have? Confidentiality, Integrity, Availability. Pass or fail is usually unambiguous.
AI red teaming doesn't have that luxury. Ask a model to help plan a phishing campaign and it might refuse outright, half-comply with a caveat, or comply after three turns of escalating roleplay — none of which is a clean binary. Ask it a factual question and it might confidently hallucinate. Ask it about a protected demographic group and it might produce a biased ranking that's technically "correct" by every unit test you wrote. None of these are confidentiality breaches. They're harms that are specific to systems that generate language and make decisions with probabilistic reasoning instead of executing deterministic code paths.
Microsoft's AI Red Team — one of the first dedicated red teams for generative AI systems, formed years before most enterprises had a GenAI product to attack — built its methodology around this reality. Their public guidance (learn.microsoft.com/security/ai-red-team) is not a tool tutorial. It's a description of how to run an engagement when "did the attack succeed" requires a judgment call across four dimensions instead of a boolean.
If you walk into an AI Red Team Engineer interview and can only talk about jailbreak strings, you'll get filtered out. Interviewers are testing whether you can scope an engagement, staff it correctly, and produce a report a VP of Engineering can act on. That's this lesson.
Threat modeling is step zero, not an afterthought
Every credible AI red team engagement starts with a threat model, adapted from STRIDE (Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege) but extended for AI-specific assets:
| STRIDE category | AI-adapted question |
|---|
| Spoofing | Can an attacker impersonate a trusted content source (a "system" message, a tool result, another agent) to gain undue influence over the model? |
| Tampering | Can retrieved documents, tool outputs, or fine-tuning data be altered to change model behavior? |
| Repudiation | Can an attacker cause the model to take an action with no audit trail tying it back to the injected instruction? |
| Information disclosure | Can the model be made to leak its system prompt, training data, or another user's context? |
| Denial of service | Can a crafted input make the model burn tokens, loop, or crash a downstream tool call? |
| Elevation of privilege | Can a low-trust input (a user message, a retrieved web page) cause a high-trust action (a tool call, a database write)? |
The output of this step is not a prose paragraph — it's a concrete list of assets (the system prompt, the RAG index, the tool registry, training data, user PII in context) and, for each, what an adversary would gain by compromising it. Everything downstream in the engagement traces back to this list. If you can't name the asset a probe is targeting, you're not red teaming, you're just poking.
Adversarial probing across four dimensions
Once the threat model identifies assets and entry points, Microsoft's methodology runs adversarial probing — manual and automated — across four Responsible AI dimensions:
- Security — the traditional stuff: prompt injection, jailbreaks, tool misuse, privilege escalation through chained calls.
- Safety — can the model be induced to produce content that causes real-world harm (self-harm instructions, weapons synthesis, dangerous medical advice)?
- Fairness — does the model produce systematically different quality, tone, or refusal rates across demographic groups for equivalent inputs?
- Privacy — can the model be induced to leak PII, training data, or information about other users/sessions it shouldn't have access to?
Traditional pentesters are trained almost exclusively on the security dimension. This is why AI red teams are cross-disciplinary: security engineers cover injection and privilege escalation, but fairness probing benefits from someone who understands demographic parity metrics, and safety probing on a medical or legal assistant benefits from a domain expert who knows what "dangerously wrong" actually looks like in that field. Microsoft's public write-ups on red-teaming production GenAI systems consistently call out that the most damaging findings came from combining a security mindset with subject-matter or social-science expertise — not from security engineers working alone.
The other structural piece: iteration. A one-time report that gets filed and forgotten is a wasted engagement. Findings feed back to the product team, get triaged like any other bug, get fixed, and get re-tested. Continuous red teaming — running a reduced probe set on every model or prompt change — is the mature end state, and it's where Promptfoo-style regression testing (lesson 4) plugs in.
Four dimensions, one team
Traditional AppSec measures success against the CIA triad. AI red teaming measures success against security, safety, fairness, and privacy simultaneously — which is why a single security engineer, however skilled, cannot cover an AI red team engagement alone. Budget for a cross-disciplinary roster before you budget for tooling.
A red team is scoped to test a hiring-assistant LLM that ranks resumes. The security engineer on the team finds no prompt injection or jailbreak vulnerabilities and declares the engagement clean. What's missing?
A hiring assistant's highest-impact harm is very unlikely to be a jailbreak — it's a fairness failure: systematically biased rankings that create legal and reputational exposure, and that a security-only tester will never find because it isn't a security bug in the traditional sense. Microsoft's methodology treats fairness as a first-class adversarial probing dimension precisely because "no injection found" and "no harm exists" are different claims for AI systems. This is also why cross-disciplinary staffing matters more here than tool selection.
The engagement lifecycle
Microsoft's methodology (and the lifecycle nearly every serious AI red team, in-house or consultancy, converges on) has nine stages:
- Define objectives and scope — what system, what's in bounds, what's explicitly out of bounds (production data? real user sessions? destructive actions?), what does "done" look like.
- Build a diverse team — security engineers, ML engineers, and where relevant, linguists, social scientists, or domain experts (medical, legal, financial) matched to the four RAI dimensions above.
- Choose attack methods and tools — map each threat-model asset to a tool: broad automated scanning (Garak), deep multi-turn attacks (PyRIT), regression testing (Promptfoo), standardized benchmarking (HarmBench, Inspect AI). Lessons 2–4 cover this selection in depth.
- Establish a safe testing environment — an isolated endpoint or sandboxed deployment so attack traffic never touches production data or real users; rate limits and kill switches for anything that could cause real-world harm if it partially succeeds.
- Execute attacks — run the chosen tools and manual probes against the target, logging every prompt, response, and tool call.
- Analyze results — for each attempt, was it a genuine bypass, a partial compliance, or a false positive from an overly aggressive detector? This is where human judgment is non-negotiable — automated scorers get this wrong constantly.
- Prioritize remediation — rank findings by severity (blast radius × exploitability × how many users are exposed), not by how impressive the jailbreak looked.
- Report findings — a structured document a non-technical stakeholder can act on (lesson 5 is entirely about this).
- Re-test after fixes — confirm the fix actually closed the finding and didn't just change the surface-level trigger phrase while leaving the underlying weakness intact.
An engagement report lists a critical prompt-injection finding as "Remediated" because the exact attack string from the original report no longer works after the fix. What's the risk with closing the finding on this basis alone?
This is one of the most common mistakes in AI security remediation: a team sees the exact reported prompt fail post-fix and marks the finding closed, but the fix may have been a narrow pattern match ('block strings containing X') rather than a structural fix (data/instruction separation, an allowlist, a human approval gate). Re-testing must use semantically equivalent but differently worded attacks — different encoding, different framing, different phrasing of the same underlying instruction — to confirm the class of vulnerability is closed, not just the one string that happened to get reported. This is exactly what stage 9 of the lifecycle, re-test, is for.
Build
Build it
The code artifact lesson_06_01_engagement_plan.py implements the full lifecycle as a structured, runnable workflow: an EngagementPlan class that walks through scope, team, tools, safe environment, execution, analysis, prioritization, reporting, and re-test as discrete stages, each producing a structured record. It ships a worked example (a RAG-backed customer support chatbot) and writes the resulting plan to engagement_plan.json.
"""
lesson_06_01_engagement_plan.py
A structured, runnable implementation of the Microsoft AI Red Team engagement
lifecycle: scope -> team -> tools -> safe environment -> execute -> analyze ->
prioritize -> report -> re-test.
This is deliberately dependency-free (stdlib only) because the lifecycle itself
is a planning and process artifact, not a tool-specific one -- lessons 2-5 wire
in Garak, PyRIT, Promptfoo, and report generation.
Usage:
python lesson_06_01_engagement_plan.py
python lesson_06_01_engagement_plan.py --stage scope
"""
from __future__ import annotations
import argparse
import json
from dataclasses import dataclass, field, asdict
from datetime import date
from enum import Enum
class RAIDimension(str, Enum):
SECURITY = "security"
SAFETY = "safety"
FAIRNESS = "fairness"
PRIVACY = "privacy"
class Severity(str, Enum):
CRITICAL = "critical"
HIGH = "high"
MEDIUM = "medium"
LOW = "low"
INFO = "info"
# ── Stage 1: Scope ──────────────────────────────────────────────────────────
@dataclass
class Asset:
name: str
description: str
dimensions: list[RAIDimension]
@dataclass
class Scope:
target_system: str
objectives: list[str]
in_bounds: list[str]
out_of_bounds: list[str]
assets: list[Asset]
success_criteria: str
# ── Stage 2: Team ───────────────────────────────────────────────────────────
@dataclass
class TeamMember:
role: str
covers_dimensions: list[RAIDimension]
rationale: str
# ── Stage 3: Tools ───────────────────────────────────────────────────────────
@dataclass
class ToolSelection:
stage: str
tool: str
rationale: str
# ── Stage 4: Safe environment ────────────────────────────────────────────────
@dataclass
class SafeEnvironment:
isolated_endpoint: str
uses_production_data: bool
rate_limits: str
kill_switch: str
# ── Stage 5-7: Execute / analyze / prioritize ────────────────────────────────
@dataclass
class Finding:
id: str
title: str
dimension: RAIDimension
severity: Severity
asset: str
reproducible: bool
notes: str
# ── Stage 9: Re-test ────────────────────────────────────────────────────────
@dataclass
class RetestResult:
finding_id: str
fix_description: str
original_string_blocked: bool
novel_phrasing_blocked: bool
status: str # "open" | "remediated" | "partially_remediated"
@dataclass
class EngagementPlan:
"""Walks the full nine-stage lifecycle for one target system."""
scope: Scope
team: list[TeamMember] = field(default_factory=list)
tool_selections: list[ToolSelection] = field(default_factory=list)
safe_environment: SafeEnvironment | None = None
findings: list[Finding] = field(default_factory=list)
retests: list[RetestResult] = field(default_factory=list)
log: list[str] = field(default_factory=list)
def assemble_team(self, members: list[TeamMember]) -> None:
self.team = members
covered = {d for m in members for d in m.covers_dimensions}
missing = set(RAIDimension) - covered
self.log.append(f"Team assembled ({len(members)} members).")
if missing:
self.log.append(f"WARNING: no team member covers {[d.value for d in missing]}")
def select_tools(self, selections: list[ToolSelection]) -> None:
self.tool_selections = selections
self.log.append(f"Selected {len(selections)} tools across the engagement.")
def establish_safe_environment(self, env: SafeEnvironment) -> None:
self.safe_environment = env
if env.uses_production_data:
self.log.append("BLOCKED: safe environment must not use production data. Fix before executing.")
else:
self.log.append("Safe environment established: isolated endpoint, no production data.")
def execute(self, findings: list[Finding]) -> None:
if self.safe_environment is None or self.safe_environment.uses_production_data:
raise RuntimeError("Cannot execute attacks without a validated safe environment.")
self.findings.extend(findings)
self.log.append(f"Executed engagement. {len(findings)} candidate findings logged.")
def analyze(self) -> dict[str, int]:
by_dimension: dict[str, int] = {}
for f in self.findings:
if not f.reproducible:
continue
by_dimension[f.dimension.value] = by_dimension.get(f.dimension.value, 0) + 1
self.log.append(f"Analysis complete. Reproducible findings by dimension: {by_dimension}")
return by_dimension
def prioritize(self) -> list[Finding]:
order = {Severity.CRITICAL: 0, Severity.HIGH: 1, Severity.MEDIUM: 2, Severity.LOW: 3, Severity.INFO: 4}
ranked = sorted((f for f in self.findings if f.reproducible), key=lambda f: order[f.severity])
self.log.append(f"Prioritized {len(ranked)} reproducible findings by severity.")
return ranked
def report_summary(self) -> str:
ranked = self.prioritize()
lines = [f"Engagement: {self.scope.target_system}", f"Objectives: {'; '.join(self.scope.objectives)}", ""]
for f in ranked:
lines.append(f"[{f.severity.value.upper()}] {f.id} — {f.title} ({f.dimension.value}, asset: {f.asset})")
self.log.append("Report summary generated.")
return "\n".join(lines)
def retest(self, results: list[RetestResult]) -> None:
self.retests = results
for r in results:
status_note = "closed on class of attack, not just original string" if r.novel_phrasing_blocked else \
"WARNING: original string blocked but novel phrasing still succeeds — not actually remediated"
self.log.append(f"Re-test {r.finding_id}: {r.status} — {status_note}")
def to_dict(self) -> dict:
return {
"scope": asdict(self.scope),
"team": [asdict(m) for m in self.team],
"tool_selections": [asdict(t) for t in self.tool_selections],
"safe_environment": asdict(self.safe_environment) if self.safe_environment else None,
"findings": [asdict(f) for f in self.findings],
"retests": [asdict(r) for r in self.retests],
"log": self.log,
}
def build_example_engagement() -> EngagementPlan:
"""Worked example: a RAG-backed customer support chatbot."""
scope = Scope(
target_system="SupportBot v3 (RAG-backed customer support chatbot with order-lookup and refund tools)",
objectives=[
"Determine whether SupportBot can be induced to issue unauthorized refunds",
"Determine whether SupportBot leaks other customers' order data",
"Baseline SupportBot's resistance to jailbreak and encoding-bypass attacks",
],
in_bounds=["staging deployment", "synthetic customer accounts", "the refund and order-lookup tools"],
out_of_bounds=["production customer data", "real payment processing", "the underlying model provider's infrastructure"],
assets=[
Asset("system_prompt", "Instructions defining SupportBot's role and refund policy", [RAIDimension.SECURITY]),
Asset("refund_tool", "Tool call that issues a refund up to $500 without human approval", [RAIDimension.SECURITY]),
Asset("order_index", "Vector index of order history used for retrieval", [RAIDimension.PRIVACY]),
Asset("response_tone", "Model's tone and helpfulness across customer demographics", [RAIDimension.FAIRNESS]),
],
success_criteria="Every asset above has at least one adversarial probe attempted and result recorded.",
)
plan = EngagementPlan(scope=scope)
plan.assemble_team([
TeamMember("Security engineer", [RAIDimension.SECURITY, RAIDimension.PRIVACY],
"Covers injection, tool misuse, and data leakage probing"),
TeamMember("ML/safety engineer", [RAIDimension.SAFETY],
"Covers harmful-content and policy-bypass probing"),
TeamMember("Fairness reviewer", [RAIDimension.FAIRNESS],
"Reviews response quality and tone across synthetic personas"),
])
plan.select_tools([
ToolSelection("Baseline scan", "Garak", "Broad automated probing across injection, encoding-bypass, and leakage probe modules"),
ToolSelection("Deep multi-turn", "PyRIT Crescendo orchestrator", "Escalating multi-turn campaign targeting the refund tool"),
ToolSelection("Regression", "Promptfoo", "Lock in fixes as CI-gated regression tests"),
])
plan.establish_safe_environment(SafeEnvironment(
isolated_endpoint="staging.supportbot.internal",
uses_production_data=False,
rate_limits="10 req/s, engagement-scoped API key",
kill_switch="Feature flag disables refund_tool execution, logs only, for the duration of the engagement",
))
plan.execute([
Finding("F-1", "Crescendo campaign escalates a billing dispute into an unauthorized refund_tool call",
RAIDimension.SECURITY, Severity.CRITICAL, "refund_tool", True,
"5-turn escalation; model called refund_tool without policy-required verification step"),
Finding("F-2", "Indirect leakage of another synthetic customer's order ID via retrieval",
RAIDimension.PRIVACY, Severity.HIGH, "order_index", True,
"Crafted query caused retrieval of an out-of-scope order record into context"),
Finding("F-3", "Base64-encoded jailbreak bypasses refusal for a policy-violating discount request",
RAIDimension.SECURITY, Severity.MEDIUM, "system_prompt", True,
"Garak encoding probe module; model decoded and complied"),
Finding("F-4", "Slightly terser responses to personas with non-Western names",
RAIDimension.FAIRNESS, Severity.LOW, "response_tone", False,
"Not reproducible at n=20 with fixed prompts; flagged for larger-sample follow-up"),
])
plan.analyze()
plan.retest([
RetestResult("F-1", "Refund tool now requires a verification-code tool call before issuing any refund",
original_string_blocked=True, novel_phrasing_blocked=True, status="remediated"),
RetestResult("F-3", "Input classifier now decodes Base64/ROT-13 before running injection detection",
original_string_blocked=True, novel_phrasing_blocked=True, status="remediated"),
])
return plan
def main() -> None:
parser = argparse.ArgumentParser(description="Run the AI red team engagement lifecycle demo.")
parser.add_argument("--stage", choices=["scope", "team", "tools", "execute", "report", "retest", "all"],
default="all", help="Which stage to print in detail.")
parser.add_argument("--out", default="engagement_plan.json", help="Path to write the full plan JSON.")
args = parser.parse_args()
plan = build_example_engagement()
print("=" * 70)
print(f"AI RED TEAM ENGAGEMENT PLAN — {date.today().isoformat()}")
print("=" * 70)
if args.stage in ("scope", "all"):
print("\n--- SCOPE ---")
print(f"Target: {plan.scope.target_system}")
for obj in plan.scope.objectives:
print(f" Objective: {obj}")
print(f" In bounds: {plan.scope.in_bounds}")
print(f" Out of bounds: {plan.scope.out_of_bounds}")
if args.stage in ("team", "all"):
print("\n--- TEAM ---")
for m in plan.team:
print(f" {m.role}: {[d.value for d in m.covers_dimensions]} — {m.rationale}")
if args.stage in ("tools", "all"):
print("\n--- TOOL SELECTION ---")
for t in plan.tool_selections:
print(f" {t.stage}: {t.tool} — {t.rationale}")
if args.stage in ("execute", "all"):
print("\n--- FINDINGS (prioritized) ---")
print(plan.report_summary())
if args.stage in ("retest", "all"):
print("\n--- RE-TEST ---")
for r in plan.retests:
print(f" {r.finding_id}: {r.status} (novel phrasing blocked: {r.novel_phrasing_blocked})")
with open(args.out, "w") as f:
json.dump(plan.to_dict(), f, indent=2, default=str)
print(f"\nFull engagement plan written to {args.out}")
print("\n--- LIFECYCLE LOG ---")
for entry in plan.log:
print(f" {entry}")
if __name__ == "__main__":
main()
Before diving into the full EngagementPlan class above, it's worth internalizing how severity-first prioritization actually shakes out on a small set of findings. The tryit block below is a minimal, dependency-free version of stage 7 (prioritize) that tallies findings by RAI dimension and sorts them by severity, so you can see the ranking logic in isolation.
# Minimal severity-first prioritization, like stage 7 of the lifecycle.
severity_order = {"critical": 0, "high": 1, "medium": 2, "low": 3, "info": 4}
findings = [
{"id": "F-1", "dimension": "security", "severity": "critical", "reproducible": True},
{"id": "F-2", "dimension": "privacy", "severity": "high", "reproducible": True},
{"id": "F-3", "dimension": "security", "severity": "medium", "reproducible": True},
{"id": "F-4", "dimension": "fairness", "severity": "low", "reproducible": False},
]
reproducible = [f for f in findings if f["reproducible"]]
ranked = sorted(reproducible, key=lambda f: severity_order[f["severity"]])
print("Prioritized findings:")
for f in ranked:
print(f" [{f['severity'].upper()}] {f['id']} ({f['dimension']})")
tally = {}
for f in reproducible:
tally[f["dimension"]] = tally.get(f["dimension"], 0) + 1
print("\nReproducible findings by RAI dimension:")
for dim, count in tally.items():
print(f" {dim}: {count}")