Phase 6: AI Red Teaming Methodology & Tools · 70 min · Garak · Python

Garak — Automated LLM Vulnerability Scanning

Manual probing finds what you thought to test for. Garak finds what you didn't.

Hiring signal: Garak is the closest thing AI red teaming has to nmap — a baseline every engagement runs before anything else. Job postings for AI Red Team Engineer roles increasingly name it explicitly. Being able to say 'I've run Garak against a real generator, read the probe/detector architecture, and know when a hit is a false positive' is a concrete, verifiable skill that separates candidates from people who've only read about prompt injection.

What you will learn

Explain Garak's probes / generators / detectors / harness architecture and how they compose into a scan
Select probe modules appropriate to a target system from Garak's 120+ probes spanning injection, jailbreaks, encoding bypasses, data leakage, and package hallucination
Run a Garak-style CLI scan against a target generator and interpret the resulting JSON/HTML report
Identify when Garak is the right tool for an engagement stage (pre-deployment baseline) versus when PyRIT or Promptfoo is a better fit

The Problem

You've scoped the engagement (lesson 1). You have a threat model, a team, and a target. Now you need to actually attack it — and manual probing, one crafted prompt at a time, doesn't scale. A serious engagement needs to run hundreds of attack variants across a dozen categories before a human ever writes a custom probe, just to establish a baseline: is this model obviously vulnerable to things every model should already resist?

That's what NVIDIA's Garak (Generative AI Red-teaming & Assessment Kit) does. It's an open-source LLM vulnerability scanner — think of it as the nmap of AI red teaming. You point it at a target, tell it which probe categories to run, and it fires a large, structured battery of adversarial prompts and scores the responses. It won't find the sophisticated multi-turn attack a skilled human would (that's PyRIT's job, lesson 3), but it will reliably surface the vulnerabilities that a real attacker would find in the first ten minutes — and if your model fails Garak's baseline, there's no point moving on to more sophisticated testing until you fix that first.

Garak's Architecture: Probes, Generators, Detectors, Harness

Garak is built around four composable concepts:

Generators — the interface to whatever you're testing. Garak ships 23 generator backends covering providers like OpenAI, Hugging Face, Cohere, REST APIs, ggml/llama.cpp models, and NVIDIA NIM endpoints, so the same probe suite runs against a hosted API or a local checkpoint without rewriting anything.
Probes — the attack prompt generators. Each probe module is a class that produces a set of adversarial prompts targeting one behavior. Garak ships 120+ probe modules grouped into families like:
dan — DAN-style jailbreaks and roleplay-based restriction bypasses
promptinject — direct prompt injection payloads
encoding — Base64, ROT-13, and other encoding-bypass attacks that smuggle instructions past naive input filters
leakreplay — probes for training-data and system-prompt leakage
packagehallucination — checks whether the model recommends installing non-existent (and therefore squattable) software packages, a supply-chain risk unique to code-generating models
xss — whether the model can be induced to emit exploitable HTML/JS in its output
toxicity, malwaregen, continuation — content-safety and completion-based probes
Detectors — for every prompt a probe sends, a detector decides whether the response counts as a "hit" (the attack succeeded). Detectors range from simple string/regex matches to classifier-based judgments. This separation matters: the same probe can be paired with different detectors depending on what "success" means for your context.
Harness — the orchestrator that runs the probe × generator × detector matrix and aggregates results into a report.

This separation of concerns is what makes Garak extensible: you can write a new probe without touching detector logic, point an existing probe at a new generator without writing new attack prompts, or swap in a stricter detector without changing what gets sent to the model.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Running a Scan, When Garak Is the Right Tool, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.