Phase 3: LLM-based Generation · ~45 minutes · Python

Evaluation — FID, CLIP Score, Human Preference

Every generative model leaderboard cites FID, CLIP score, and a win rate from a human-preference arena. Each number has a failure mode a determined researcher can game. If you do not know the failure modes, you cannot tell a real improvement from a gaming run.

What you will learn

Implement evaluation — fid, clip score, human preference from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Build
Languages: Python
Prerequisites: Generative Models — Taxonomy & History, Decision Trees and Random Forests (Core ML: Algorithms from Scratch)
Time: ~45 minutes

The Problem

A generative model is judged on sample quality and conditioning adherence. Neither has a closed-form measure. Your model has to render 10,000 images; something has to assign them numbers; you have to trust the numbers across model families, across resolutions, across architectures. Three metrics survived the 2014-2026 gauntlet:

FID (Fréchet Inception Distance). A distance between two distributions — real and generated — in an Inception network's feature space. Lower is better.
CLIP score. Cosine similarity between a generated image's CLIP-image embedding and a prompt's CLIP-text embedding. Higher is better. Measures prompt adherence.
Human preference. Pit two models head-to-head on the same prompt, have humans (or a GPT-4-class model) pick the better one, aggregate to an Elo score.

You will also see: IS (inception score, largely retired), KID, CMMD, ImageReward, PickScore, HPSv2, MJHQ-30k. Each corrects for one failure of the previous.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.