Phase 3: Vision-Language Model · ~90 minutes · Python

Multimodal Evaluation

Training is half the loop.

Hiring signal: Can build multimodal evaluation end to end

What you will learn

Compute Recall@K from a similarity matrix between image and caption embeddings.
Compute exact-match VQA accuracy from a model that maps (image, question) pairs to a fixed answer vocabulary.
Compute BLEU-4 from generated and reference token sequences without any external library.
Run all three evals against a synthetic suite built on top of the trained model from lesson 62.

Introduction

The other half is measurement. This lesson builds three evaluation surfaces from primitives: image-caption retrieval reported as R@1, R@5, R@10; visual question answering reported as exact match accuracy; and image captioning reported as BLEU-4. Each metric is a function over the model's outputs and a synthetic eval suite that runs in seconds.

Type: Build Languages: Python Prerequisites: Phase 19 lessons 58-62 (Track E foundations: encoder, transformer, projection, cross-attention fusion, pretraining) Time: ~90 minutes

Objective

Learning objectives

Compute Recall@K from a similarity matrix between image and caption embeddings.
Compute exact-match VQA accuracy from a model that maps (image, question) pairs to a fixed answer vocabulary.
Compute BLEU-4 from generated and reference token sequences without any external library.
Run all three evals against a synthetic suite built on top of the trained model from lesson 62.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Token