Phase 9: Evaluation, Production & Capstone · 50 min · Python · fal.ai API · CLIP

Production QA: Generate-N-Pick-Best

Generate 8 candidates, score them, deliver the best. That's the generate-N-pick-best strategy that makes generative media production-ready.

Hiring signal: Production QA (generate-N-pick-best, quality gates, human-in-the-loop) is the difference between a demo and a production system — interviewers test for this.

What you will learn

Implement generate-N-pick-best: generate 4-8 candidates, score them, deliver the highest-scoring output
Configure automated quality gates: reject outputs below quality threshold before delivery
Set quality gate thresholds: CLIPScore, Aesthetic Score, artifact level
Implement human-in-the-loop review: routing low-confidence outputs to human reviewers

The Problem

A single image generation is a lottery — sometimes you get a great image, sometimes you get garbage. In production, you can't deliver garbage to users.

Generate-N-Pick-Best solves this:

Generate 4-8 candidates with different seeds
Score each candidate with automated metrics (CLIPScore, Aesthetic, ImageReward)
Deliver the highest-scoring candidate
If none pass quality gates, either regenerate or route to human review

This is the difference between a demo and a production system.

What you'll build
Implement generate-N-pick-best with automated scoring, configurable quality gates (thresholds), and human-in-the-loop review for low-confidence outputs. Build a production QA pipeline that ensures every delivered asset meets quality standards.

Generate-N-Pick-Best

The Strategy

import asyncio
from dataclasses import dataclass

@dataclass
class Candidate:
    """A single generation candidate."""
    id: str
    seed: int
    image_url: str
    clip_score: float = 0.0
    aesthetic_score: float = 0.0
    image_reward: float = 0.0
    total_score: float = 0.0

async def generate_candidates(prompt: str, n: int = 8,
                               model: str = "fal-ai/flux/schnell") -> list[Candidate]:
    """Generate N candidates with different seeds."""
    candidates = []
    
    async def generate_one(seed: int) -> Candidate:
        # In production: call fal.ai or RunPod
        # result = await fal.client.subscribe_async(model, arguments={
        #     "prompt": prompt, "seed": seed, ...
        # })
        # Simulated for demo
        await asyncio.sleep(0.1)
        return Candidate(
            id=f"cand-{seed}",
            seed=seed,
            image_url=f"https://cdn.example.com/img_{seed}.png",
        )
    
    # Generate all N candidates in parallel
    tasks = [generate_one(seed) for seed in range(42, 42 + n)]
    candidates = await asyncio.gather(*tasks)
    return list(candidates)

async def score_candidates(candidates: list[Candidate], prompt: str) -> list[Candidate]:
    """Score each candidate with quality metrics."""
    for candidate in candidates:
        # In production: compute CLIPScore, Aesthetic, ImageReward
        # Simulated
        import random
        candidate.clip_score = 20 + random.uniform(0, 20)       # 20-40
        candidate.aesthetic_score = 4 + random.uniform(0, 5)    # 4-9
        candidate.image_reward = -0.5 + random.uniform(0, 2)    # -0.5 to 1.5
        
        # Weighted total (normalize to 0-10 scale)
        candidate.total_score = (
            (candidate.clip_score / 10) * 0.3 +      # CLIPScore 0-100 → 0-10
            candidate.aesthetic_score * 0.4 +          # Aesthetic 1-10
            ((candidate.image_reward + 1) / 0.3) * 0.3  # ImageReward -1 to 2 → 0-10
        )
    
    # Sort by total score (highest first)
    candidates.sort(key=lambda c: c.total_score, reverse=True)
    return candidates

async def generate_and_pick_best(prompt: str, n: int = 8) -> dict:
    """Full generate-N-pick-best pipeline."""
    # 1. Generate N candidates
    candidates = await generate_candidates(prompt, n=n)
    
    # 2. Score all candidates
    candidates = await score_candidates(candidates, prompt)
    
    # 3. Pick the best
    best = candidates[0]
    
    return {
        "prompt": prompt,
        "candidates_generated": n,
        "best_candidate": {
            "id": best.id,
            "seed": best.seed,
            "image_url": best.image_url,
            "clip_score": round(best.clip_score, 2),
            "aesthetic_score": round(best.aesthetic_score, 2),
            "image_reward": round(best.image_reward, 3),
            "total_score": round(best.total_score, 2),
        },
        "all_scores": [
            {"id": c.id, "seed": c.seed, "total": round(c.total_score, 2)}
            for c in candidates
        ],
    }

Why does generating multiple candidates and picking the best produce higher quality than generating a single image?

It costs less

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Quality Gates, Human-in-the-Loop Review, Cost Analysis, Key Takeaways, What's Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Related lessons

Browse all courses · View pricing · DeVenture Academy