Phase 3: Vision-Language Model · ~90 minutes · Python

Vision-Language Pretraining

The encoder, projection, and decoder are wired.

Hiring signal: Can build vision-language pretraining end to end

What you will learn

Implement InfoNCE contrastive loss across a batch of image-caption pairs.
Compose contrastive loss with autoregressive language modeling loss.
Synthesize a 200-pair mock image-caption corpus with no real dataset download.
Run a 50-step demo training loop and observe both losses decreasing.

Introduction

Now train them together. Two objectives drive learning: a contrastive image-text loss (InfoNCE) that pulls matching pairs together in the joint embedding space, and a language modeling loss that asks the decoder to caption each image. Combined, they teach the network both to find the right image for a caption and to write a caption for the image.

Type: Build Languages: Python Prerequisites: Phase 19 lessons 30-37 (Track B foundations) Time: ~90 minutes

Objective

Learning objectives

Implement InfoNCE contrastive loss across a batch of image-caption pairs.
Compose contrastive loss with autoregressive language modeling loss.
Synthesize a 200-pair mock image-caption corpus with no real dataset download.
Run a 50-step demo training loop and observe both losses decreasing.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.