Phase 9: Evaluation, Production & Capstone · 50 min · Python · OpenCV · PyTorch

Video Quality Metrics

Temporal consistency, prompt fidelity, artifact level, motion realism — video quality is multidimensional and measurable.

Hiring signal: Video quality metrics knowledge (temporal consistency, artifact level, motion realism) is a specialized skill that video-focused companies specifically test for.

What you will learn

Measure temporal consistency: frame-to-frame visual coherence (flickering, morphing, object persistence)
Measure prompt fidelity (1-10): how well video matches the text prompt
Measure aesthetic quality (1-10) and artifact level (1-10): visual appeal and distortion
Evaluate human motion realism and physics simulation accuracy

The Problem

A team generates 50 videos with Kling and 50 with Runway. Which produces better videos? Video quality is more complex than image quality because it adds a temporal dimension:

Temporal consistency: do objects stay consistent across frames? (no flickering, morphing)
Prompt fidelity: does the video match the text prompt?
Artifact level: are there visual distortions? (warping, blurring, color shifts)
Motion realism: does motion look natural? (physics, human motion)

What you'll build
Implement temporal consistency measurement (frame-to-frame coherence), prompt fidelity scoring, artifact detection, motion realism evaluation, and a multi-dimensional video quality evaluator using VBench and EvalCrafter frameworks.

Temporal Consistency

What It Measures

Temporal consistency measures whether visual elements stay coherent across frames:

Flickering: brightness/color changes between frames
Morphing: objects changing shape unexpectedly
Object persistence: do objects stay the same throughout the video?

import cv2
import numpy as np

def measure_temporal_consistency(video_path: str) -> dict:
    """Measure temporal consistency metrics."""
    cap = cv2.VideoCapture(video_path)
    frames = []
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break
        frames.append(frame)
    cap.release()
    
    if len(frames) < 2:
        return {"error": "Need at least 2 frames"}
    
    # 1. Frame-to-frame brightness consistency
    brightness_scores = []
    for i in range(1, len(frames)):
        prev_gray = cv2.cvtColor(frames[i-1], cv2.COLOR_BGR2GRAY)
        curr_gray = cv2.cvtColor(frames[i], cv2.COLOR_BGR2GRAY)
        prev_mean = np.mean(prev_gray)
        curr_mean = np.mean(curr_gray)
        diff = abs(prev_mean - curr_mean) / 255.0
        brightness_scores.append(1.0 - diff)
    
    # 2. Structural similarity (SSIM) between consecutive frames
    ssim_scores = []
    for i in range(1, len(frames)):
        prev_gray = cv2.cvtColor(frames[i-1], cv2.COLOR_BGR2GRAY)
        curr_gray = cv2.cvtColor(frames[i], cv2.COLOR_BGR2GRAY)
        # Use optical flow to align frames before comparing
        flow = cv2.calcOpticalFlowFarneback(
            prev_gray, curr_gray, None, 0.5, 3, 15, 3, 5, 1.2, 0)
        warped = cv2.remap(prev_gray, flow, None, cv2.INTER_LINEAR)
        score = ssim(prev_gray, warped)
        ssim_scores.append(score)
    
    return {
        "brightness_consistency": np.mean(brightness_scores),
        "structural_consistency": np.mean(ssim_scores),
        "temporal_score": (np.mean(brightness_scores) + np.mean(ssim_scores)) / 2,
        "frame_count": len(frames),
    }

def ssim(img1: np.ndarray, img2: np.ndarray) -> float:
    """Compute Structural Similarity Index."""
    from skimage.metrics import structural_similarity
    return structural_similarity(img1, img2, data_range=255)

Temporal Consistency Scale (1-10)

Score	Quality	Description
8-10	Excellent	Smooth, no flickering, objects persistent
6-8	Good	Minor flickering, mostly consistent
4-6	Fair	Noticeable flickering, some morphing
2-4	Poor	Significant flickering, objects morph
1-2	Bad	Severe inconsistency, unwatchable

Why is temporal consistency critical for video generation but not for image generation?

Temporal consistency affects image quality too

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Prompt Fidelity, Artifact Level, Motion Realism, VBench: Comprehensive Video Benchmark, Multi-Dimensional Video Quality Evaluator, Key Takeaways, What's Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.