Phase 9: Evaluation, Production & Capstone · 50 min · Python · Hugging Face · C2PA

Content Moderation & Safety

Prompt filtering, output moderation, C2PA provenance, SynthID watermarking — safety is a production requirement, not a feature.

Real-world application: Content moderation and safety engineering (NSFW checkers, C2PA, watermarking) demonstrates you understand the compliance and safety requirements of production generative media.

What you will learn

Implement prompt filtering: blocking prohibited content before generation (violence, NSFW, PII)
Implement output moderation: scanning generated images/videos for prohibited content
Use NSFW safety checkers: Stable Diffusion safety checker, custom classifiers
Implement C2PA content provenance metadata and watermarking (SynthID)

Objective

Judge creative outputs with both measurable gates and human critique, then defend the release decision. In the paired studio lab, build a content safety pipeline that: (1) filters prompts for prohibited content before generation, (2) scans generated images with a safety checker (NSFW detection), (3) embeds C2PA provenance metadata into the generated image, (4) logs all safety decisions. Test with both safe and edge-case prompts.

The Problem

A generative media platform lets users type any prompt. Without safety:

Users generate NSFW content → platform gets banned
Users generate violent content → legal liability
Users generate deepfakes → reputational damage
No provenance → can't distinguish AI-generated from real

Content moderation and safety engineering is a production requirement:

Prompt filtering: block prohibited content before generation
Output moderation: scan generated images/videos for prohibited content
NSFW checkers: Stable Diffusion safety checker, custom classifiers
C2PA provenance: metadata proving AI origin
Watermarking: SynthID invisible watermarks

What you'll build
Implement prompt filtering (block prohibited content), output moderation (scan generated images), NSFW safety checkers, C2PA content provenance metadata, and SynthID-style watermarking for AI-generated content.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.