The Problem
You have spent eight phases building real artifacts: a multi-layer prompt injection detector, a full agent security audit with a threat model, a red team engagement report against a RAG system using Garak and PyRIT, and a production security architecture with a monitoring dashboard and an incident response playbook. Almost every AI security candidate applying for the same roles has a resume that says "familiar with prompt injection" and a LinkedIn certificate badge. Almost none of them have a document a hiring manager can actually read.
That gap is the opportunity. AI security is young enough that there is no accepted credential that proves competence the way a CPA license proves accounting competence. Hiring managers know this, so they fall back on the one signal they trust: work product. A threat model you wrote for a real (or realistic) multi-agent system tells them more in five minutes than a resume tells them in five minutes plus a 30-minute phone screen. This is the same reason security consultancies hire based on sample deliverables, not just interviews — the deliverable is the job.
The mistake most candidates make is treating their course projects as homework to check off, then writing a generic "completed AI security training" line on their resume. The fix is to treat every project you built in Phases 2, 4, 6, and 8 as a work sample from day one — which means writing it the way you'd write it for a client, not a grader.
Redact, don't delete
The instinct when preparing a real engagement writeup for public consumption is to delete anything sensitive. Don't delete — redact and generalize. "NimbusDesk Support Platform" becomes "a Series B fintech support platform." A real CVSS score, attack chain, and remediation timeline stay intact. The value of the case study is the methodology and the result; the identifying details are the only thing you strip out.
What makes an artifact a portfolio piece, not homework
Four things separate a case study a hiring manager will read from one they'll skim past:
| Homework version | Portfolio version |
|---|
| "Implemented a prompt injection detector as part of Phase 2" | "Built a 3-layer detector achieving 94% detection rate at 3% false-positive rate against a 240-example adversarial test set, cutting successful bypasses from 17/20 known techniques to 2/20" |
| Full code dump, no narrative | Executive summary first, then methodology, then the code as supporting evidence |
| "Found some vulnerabilities in a sample agent" | Severity-ranked findings with CVSS-style scoring, reproduction steps, and a specific remediation for each |
| Screenshot of a passing test suite | A before/after re-test table showing the exact attack chain that used to succeed and now doesn't |
Your four course projects map directly onto the four artifact types AI security hiring managers ask for by name in job postings and take-home exercises:
- Phase 2 — Injection Detector + Jailbreak Library → a working detector repo plus a documented library of 20+ jailbreak techniques with measured bypass rates. This is the artifact that answers "show me you understand OWASP LLM01 beyond the definition."
- Phase 4 — Agent Security Audit → a threat model plus a severity-ranked vulnerability report with implemented fixes. This is the artifact that answers the "review this agent codebase" interview test almost every AI security team runs.
- Phase 6 — Full Red Team Engagement Report → a Garak + PyRIT engagement with methodology, findings, attack chains, and re-test results. This is the closest thing to a real consulting deliverable you can put in front of a hiring manager.
- Phase 8 — Production AI Security System → an end-to-end architecture with a monitoring dashboard, CI/CD security gates, and an incident response playbook. This is the artifact that proves you can operate security, not just find vulnerabilities once.
Structuring a case study
Every case study should follow the same five-part shape, regardless of which project it's built from:
- One-line outcome — what changed, quantified. "Reduced successful indirect injection attempts from 41% to 6% across a 50-attack test suite."
- Context — one paragraph on the system under test: what it does, what made it worth attacking (RAG + tool access + multi-tenant, for example).
- Methodology — the tools and technique categories used (Garak probes, PyRIT orchestrators, manual roleplay jailbreaks), not a line-by-line narration of every command you ran.
- Findings — severity-ranked, each with a reproduction step and a specific fix. This is the section hiring managers actually read closely.
- Evidence — the code, the report, the before/after metrics. Link to the repo; don't paste 400 lines inline.
For a threat model specifically, the audience often isn't technical — a hiring manager screening resumes, or in the real job, a product lead who needs to sign off on a launch. Presenting it well means leading with the two or three highest-blast-radius findings in plain language before the STRIDE/PASTA table, not after it.
You're preparing your Phase 6 red team engagement report for your portfolio. The report was written against a fictional "Aurora Health" chatbot you built for the course, so there's no real client to protect. What should you still redact or generalize before publishing it publicly?
Even a fictional target's engagement report can leak real operational details — a staging API key, a real hostname you happened to use for convenience, a snippet of your actual test infrastructure. Redaction isn't about protecting a client that doesn't exist here; it's about not leaking your own credentials or infrastructure details into a public repo. Severity ratings and tool names should stay — they're exactly the signal a hiring manager is looking for, and hiding them makes the report weaker, not more professional.
What to redact vs. what to show
A concrete checklist, in order of how often engineers get this wrong:
Always redact:
- Real company/client names → replace with a generic descriptor ("a Series B fintech," "a healthcare RAG chatbot")
- API keys, tokens, credentials — even test/staging ones, even if expired
- Internal hostnames, IP addresses, VPN endpoints
- Any PII that appeared in test data or logs
Always show:
- The methodology and tools used
- Severity ratings and CVSS-style scoring
- The attack chain and reproduction steps (generalized to the redacted system)
- Before/after metrics — this is the single most persuasive thing in the document
- Your own code, in full
Judgment call — usually show, sometimes trim:
- Full transcripts of jailbreak attempts (show 2-3 representative ones, summarize the rest)
- Screenshots of dashboards (show one annotated example, not fifteen near-duplicates)
A hiring manager gives you 3 minutes with your portfolio during a screening call. You have all four course projects ready. What's the highest-leverage thing to lead with?
Three minutes is not enough time to walk through four projects — it's enough time to make one strong impression. Lead with whatever most directly matches the specific role: an AI Red Team Engineer posting should see your red team engagement report first; a DevSecOps for AI posting should see your production security system and CI/CD gates first. The quantified one-liner is what makes them ask a follow-up question instead of nodding and moving on — that follow-up question is the interview actually starting.
Build
Build it
The PortfolioBuilder script below scans a directory of your own project artifacts (the .md reports and .py code files you produced in Phases 2, 4, 6, and 8), classifies each one against the four course projects, extracts a title, summary, and any quantified metrics it finds, and runs a redaction pass that flags emails, API-key-shaped strings, IP addresses, internal hostnames, and named-client references before you publish anything. It writes a PORTFOLIO.md you can drop straight into a GitHub repo README, plus a portfolio_index.json for programmatic use (e.g., rendering it on a personal site).
Run it with no arguments and it builds and scans a small bundled demo tree so you can see the output immediately; point --dir at your own project folders when you're ready to build the real thing.
"""
lesson_09_01_portfolio_builder.py
Scans a directory of your own past project artifacts (markdown reports, threat
models, Python detectors/guardrails) and generates a structured portfolio
index (PORTFOLIO.md + portfolio_index.json) with one case-study summary per
artifact — plus a redaction pass that flags content you should not publish
as-is.
This is built to run directly against the 4 course project deliverables:
Phase 2: injection detector + jailbreak library
Phase 4: agent security audit (threat model + vulnerability report)
Phase 6: full red team engagement report (Garak + PyRIT)
Phase 8: production AI security system (architecture + IR playbook)
Usage:
python lesson_09_01_portfolio_builder.py --dir /path/to/your/projects
python lesson_09_01_portfolio_builder.py # runs on a bundled demo tree
Requirements:
Standard library only. No API keys, no network calls, runs fast on CPU.
"""
from __future__ import annotations
import argparse
import json
import re
import tempfile
from dataclasses import dataclass, field, asdict
from pathlib import Path
from datetime import date
# ── Redaction heuristics ──────────────────────────────────────────────────────
# These are the patterns that get engineers in trouble when they publish a real
# engagement writeup: internal hostnames, emails, credential-shaped strings,
# and IP addresses. This is a heuristic pass, not a guarantee — always do a
# manual read before you publish anything from a real client engagement.
REDACTION_PATTERNS: list[tuple[str, str]] = [
(r"\b[A-Za-z0-9._%+-]+@(?!example\.com)[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b", "email address"),
(r"\b(?:sk|pk|api|key)[-_][A-Za-z0-9_-]{10,}\b", "API-key-shaped string"),
(r"\b(?:\d{1,3}\.){3}\d{1,3}\b", "IPv4 address"),
(r"\b(?:internal|corp|vpn)[-.][A-Za-z0-9.-]+\.(?:com|net|local)\b", "internal hostname"),
(r"(?i)\bclient[:\s]+[A-Z][A-Za-z0-9 &]{2,40}\b", "named client reference"),
(r"(?i)\b(?:password|secret|token)\s*[:=]\s*\S+", "inline credential"),
]
# ── Artifact classification ───────────────────────────────────────────────────
# Maps filename/heading keywords to the 4 course projects so the generated
# portfolio groups artifacts the way a hiring manager expects to see them.
PROJECT_TAGS: dict[str, tuple[str, str]] = {
"injection": ("Prompt Injection Detector & Jailbreak Library", "Phase 2"),
"jailbreak": ("Prompt Injection Detector & Jailbreak Library", "Phase 2"),
"agent_audit": ("Agent Security Audit", "Phase 4"),
"agent-audit": ("Agent Security Audit", "Phase 4"),
"threat_model": ("Agent Security Audit", "Phase 4"),
"threat-model": ("Agent Security Audit", "Phase 4"),
"red_team": ("Full Red Team Engagement Report", "Phase 6"),
"red-team": ("Full Red Team Engagement Report", "Phase 6"),
"garak": ("Full Red Team Engagement Report", "Phase 6"),
"pyrit": ("Full Red Team Engagement Report", "Phase 6"),
"production_security": ("Production AI Security System", "Phase 8"),
"production-security": ("Production AI Security System", "Phase 8"),
"guardrail": ("Production AI Security System", "Phase 8"),
"incident_response": ("Production AI Security System", "Phase 8"),
}
ARTIFACT_EXTENSIONS = {".md", ".py"}
@dataclass
class RedactionFlag:
line_no: int
pattern_label: str
snippet: str
@dataclass
class CaseStudy:
path: str
title: str
project: str
project_phase: str
kind: str # "report" | "code"
summary: str
metrics: list[str] = field(default_factory=list)
redaction_flags: list[RedactionFlag] = field(default_factory=list)
word_count: int = 0
def to_dict(self) -> dict:
d = asdict(self)
return d
# ── Scanning ───────────────────────────────────────────────────────────────────
def classify_project(path: Path) -> tuple[str, str]:
haystack = path.stem.lower().replace(" ", "_")
for keyword, (project, phase) in PROJECT_TAGS.items():
if keyword.replace("-", "_") in haystack.replace("-", "_"):
return project, phase
return "Uncategorized artifact", "—"
def extract_title(text: str, fallback: str) -> str:
for line in text.splitlines():
stripped = line.strip()
if stripped.startswith("# "):
return stripped[2:].strip()
return fallback
def extract_summary(text: str, kind: str) -> str:
if kind == "code":
# Use the module docstring's first non-empty line as the summary.
m = re.search(r'"""(.*?)"""', text, re.DOTALL)
if m:
for line in m.group(1).strip().splitlines():
if line.strip():
return line.strip()
return "No module docstring found — add one so this reads as a case study, not a script."
# Markdown: first paragraph after the title, skipping headings/blockquotes.
lines = text.splitlines()
para: list[str] = []
seen_title = False
for line in lines:
s = line.strip()
if not seen_title and s.startswith("# "):
seen_title = True
continue
if not seen_title:
continue
if not s:
if para:
break
continue
if s.startswith("#") or s.startswith(">"):
continue
para.append(s)
return " ".join(para)[:400] if para else "No summary paragraph found under the title."
METRIC_PATTERN = re.compile(
r"\b\d{1,3}(?:\.\d+)?\s?%(?:\s+\w+){0,4}" # "94% detection rate"
r"|\b\d+\s*/\s*\d+\s+(?:findings|checks|probes|techniques)" # "17/20 techniques"
r"|severity[:\s]+(?:critical|high|medium|low)",
re.IGNORECASE,
)
def extract_metrics(text: str) -> list[str]:
return sorted(set(m.strip() for m in METRIC_PATTERN.findall(text)))[:6]
def scan_redactions(text: str) -> list[RedactionFlag]:
flags: list[RedactionFlag] = []
for line_no, line in enumerate(text.splitlines(), start=1):
for pattern, label in REDACTION_PATTERNS:
if re.search(pattern, line):
flags.append(RedactionFlag(line_no=line_no, pattern_label=label, snippet=line.strip()[:80]))
return flags
def build_case_study(path: Path, root: Path) -> CaseStudy:
text = path.read_text(encoding="utf-8", errors="ignore")
kind = "code" if path.suffix == ".py" else "report"
project, phase = classify_project(path)
title = extract_title(text, fallback=path.stem.replace("_", " ").replace("-", " ").title())
return CaseStudy(
path=str(path.relative_to(root)),
title=title,
project=project,
project_phase=phase,
kind=kind,
summary=extract_summary(text, kind),
metrics=extract_metrics(text),
redaction_flags=scan_redactions(text),
word_count=len(text.split()),
)
def scan_directory(root: Path) -> list[CaseStudy]:
case_studies = []
for path in sorted(root.rglob("*")):
if path.is_file() and path.suffix in ARTIFACT_EXTENSIONS:
case_studies.append(build_case_study(path, root))
return case_studies
# ── Rendering ──────────────────────────────────────────────────────────────────
def render_portfolio_md(case_studies: list[CaseStudy]) -> str:
lines = [
"# AI Security Portfolio",
"",
f"_Generated {date.today().isoformat()} from {len(case_studies)} project artifact(s)._",
"",
"Each entry below maps to a course project. Redaction flags must be resolved",
"before publishing — see the checklist at the bottom.",
"",
]
by_project: dict[str, list[CaseStudy]] = {}
for cs in case_studies:
by_project.setdefault(cs.project, []).append(cs)
for project, items in by_project.items():
phase = items[0].project_phase
lines.append(f"## {project} ({phase})")
lines.append("")
for cs in items:
lines.append(f"### {cs.title}")
lines.append(f"`{cs.path}` — {cs.word_count} words")
lines.append("")
lines.append(cs.summary or "_(no summary extracted — add a lead paragraph)_")
lines.append("")
if cs.metrics:
lines.append("**Measured results:** " + ", ".join(cs.metrics))
lines.append("")
if cs.redaction_flags:
lines.append(f"**Redaction review needed** — {len(cs.redaction_flags)} flag(s) before this can be published:")
for flag in cs.redaction_flags[:5]:
lines.append(f"- line {flag.line_no} ({flag.pattern_label}): `{flag.snippet}`")
lines.append("")
lines.append("## Publishing checklist")
lines.append("")
lines.append("- [ ] Every redaction flag above has been reviewed and resolved")
lines.append("- [ ] Real client/company names are replaced with a generic label (\"a Series B fintech\")")
lines.append("- [ ] Every finding has a reproduction step a reader could actually run")
lines.append("- [ ] Each case study leads with the outcome, not the process")
lines.append("- [ ] Code artifacts run standalone without your original test infrastructure")
return "\n".join(lines)
# ── Demo tree (used when no --dir is supplied) ────────────────────────────────
DEMO_ARTIFACTS: dict[str, str] = {
"injection_detector_report.md": """# Multi-Layer Prompt Injection Detector
Built a three-layer detector (regex heuristics, scoring, DeBERTa-style classifier)
against a 240-example internal test set spanning direct injection, indirect
injection, and encoding-based jailbreaks.
Result: 94% detection rate at a 3% false-positive rate on held-out examples.
Severity: high-risk bypasses reduced from 17/20 techniques succeeding to 2/20.
Contact: research-lead@internal-corp.local for the full dataset.
""",
"agent_audit_threat_model.md": """# Threat Model: Multi-Agent Support System
Reviewed an orchestrator + 3 sub-agents with RAG and 2 MCP tool servers.
Client: NimbusDesk Support Platform (redact before publishing).
Identified 6 attack vectors including tool output poisoning and planner
manipulation. 4/6 rated critical or high severity.
""",
"red_team_engagement_report.md": """# Red Team Engagement — Garak + PyRIT
Ran a structured engagement against a customer-facing RAG chatbot using Garak
(118 probes) and PyRIT (Crescendo, XPIA orchestrators).
Findings: 9 findings total, severity: critical x2, severity: high x3.
Attacker success rate on indirect injection chain: 41%.
API key used for staging: sk-test-abcdef1234567890
""",
"production_security_system.py": '''"""
Multi-layer guardrail pipeline for a production support agent: input
classification, PII detection, tool-call authorization, and structured
audit logging. Deployed behind a FastAPI gateway.
"""
def run_pipeline(user_input: str) -> dict:
return {"input": user_input, "blocked": False}
''',
}
def build_demo_tree() -> Path:
tmp = Path(tempfile.mkdtemp(prefix="portfolio_demo_"))
for filename, content in DEMO_ARTIFACTS.items():
(tmp / filename).write_text(content, encoding="utf-8")
return tmp
# ── CLI ────────────────────────────────────────────────────────────────────────
def main() -> None:
parser = argparse.ArgumentParser(description="Build a portfolio index from your project artifacts.")
parser.add_argument("--dir", type=str, default=None, help="Directory of .md/.py project artifacts to scan.")
parser.add_argument("--out", type=str, default="PORTFOLIO.md", help="Output markdown file.")
args = parser.parse_args()
if args.dir:
root = Path(args.dir).resolve()
print(f"Scanning your artifacts in {root} ...")
else:
root = build_demo_tree()
print(f"No --dir supplied. Running on a bundled demo tree: {root}")
print("Point this at your own Phase 2/4/6/8 deliverables with --dir.\\n")
case_studies = scan_directory(root)
if not case_studies:
print(f"No .md or .py artifacts found under {root}.")
return
portfolio_md = render_portfolio_md(case_studies)
out_path = Path(args.out)
out_path.write_text(portfolio_md, encoding="utf-8")
index_path = out_path.with_suffix(".json")
index_path.write_text(
json.dumps([cs.to_dict() for cs in case_studies], indent=2, default=str),
encoding="utf-8",
)
total_flags = sum(len(cs.redaction_flags) for cs in case_studies)
print(f"Scanned {len(case_studies)} artifact(s).")
print(f"Wrote {out_path} and {index_path}.")
print(f"Total redaction flags across all artifacts: {total_flags}")
if total_flags:
print("Resolve every flag before publishing — see the checklist at the bottom of PORTFOLIO.md.")
if __name__ == "__main__":
main()