Phase 6: AI Red Teaming Methodology & Tools · 60 min · Python · PyRIT · MITRE ATLAS

The Microsoft AI Red Team Methodology & Engagement Lifecycle

AI red teaming isn't pentesting with extra steps — it's probing for harms the CIA triad was never designed to catch.

Hiring signal: AI Red Team Engineer and AI Security Consultant postings screen for candidates who can run a structured engagement, not just throw jailbreaks at a chatbot. Microsoft, Anthropic, OpenAI, and Google all operate dedicated AI red teams that follow a lifecycle like this one — knowing scope-to-retest cold is what separates a hire from a candidate who 'has done some prompt injection stuff.'

What you will learn

Explain why AI red teaming targets context-specific harms (security, safety, fairness, privacy) instead of only the CIA triad, and why that changes team composition and success criteria
Walk the full Microsoft AI Red Team engagement lifecycle: scope → team → tools → safe environment → execute → analyze → prioritize → report → re-test
Design a cross-disciplinary red team roster for a given AI system and justify each role
Produce a structured engagement plan document that a security lead could approve before any attack traffic is sent

The Problem

A traditional penetration test has a clean success criterion: did you get unauthorized access, exfiltrate data, or execute code you shouldn't have? Confidentiality, Integrity, Availability. Pass or fail is usually unambiguous.

AI red teaming doesn't have that luxury. Ask a model to help plan a phishing campaign and it might refuse outright, half-comply with a caveat, or comply after three turns of escalating roleplay — none of which is a clean binary. Ask it a factual question and it might confidently hallucinate. Ask it about a protected demographic group and it might produce a biased ranking that's technically "correct" by every unit test you wrote. None of these are confidentiality breaches. They're harms that are specific to systems that generate language and make decisions with probabilistic reasoning instead of executing deterministic code paths.

Microsoft's AI Red Team — one of the first dedicated red teams for generative AI systems, formed years before most enterprises had a GenAI product to attack — built its methodology around this reality. Their public guidance (learn.microsoft.com/security/ai-red-team) is not a tool tutorial. It's a description of how to run an engagement when "did the attack succeed" requires a judgment call across four dimensions instead of a boolean.

If you walk into an AI Red Team Engineer interview and can only talk about jailbreak strings, you'll get filtered out. Interviewers are testing whether you can scope an engagement, staff it correctly, and produce a report a VP of Engineering can act on. That's this lesson.

Threat Modeling Is Step Zero, Not an Afterthought

Every credible AI red team engagement starts with a threat model, adapted from STRIDE (Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege) but extended for AI-specific assets:

STRIDE category	AI-adapted question
Spoofing	Can an attacker impersonate a trusted content source (a "system" message, a tool result, another agent) to gain undue influence over the model?
Tampering	Can retrieved documents, tool outputs, or fine-tuning data be altered to change model behavior?
Repudiation	Can an attacker cause the model to take an action with no audit trail tying it back to the injected instruction?
Information disclosure	Can the model be made to leak its system prompt, training data, or another user's context?
Denial of service	Can a crafted input make the model burn tokens, loop, or crash a downstream tool call?
Elevation of privilege	Can a low-trust input (a user message, a retrieved web page) cause a high-trust action (a tool call, a database write)?

The output of this step is not a prose paragraph — it's a concrete list of assets (the system prompt, the RAG index, the tool registry, training data, user PII in context) and, for each, what an adversary would gain by compromising it. Everything downstream in the engagement traces back to this list. If you can't name the asset a probe is targeting, you're not red teaming, you're just poking.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Adversarial Probing Across Four Dimensions, The Engagement Lifecycle, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Red Teaming