Phase 4: Vision Transformers · ~60 minutes · Python

SAM 3 & Open-Vocabulary Segmentation

Give a model a text prompt and an image and get masks for every matching object. SAM 3 made that a single forward pass.

What you will learn

Distinguish SAM (visual prompts only), Grounded SAM / SAM 2 (detector + SAM), and SAM 3 (native text prompts via Promptable Concept Segmentation)
Explain the SAM 3 architecture: shared backbone + image detector + memory-based video tracker + presence head + decoupled detector-tracker design
Use Hugging Face `transformers` SAM 3 integration for text-prompted detection, segmentation, and video tracking
Pick between SAM 3, Grounded SAM 2, YOLO-World, and SAM-MI based on latency, concept complexity, and deployment target

Introduction

Type: Use + Build
Languages: Python
Prerequisites: Semantic Segmentation — U-Net, Instance Segmentation — Mask R-CNN, Open-Vocabulary Vision — CLIP
Time: ~60 minutes

Objective

Learning objectives

Distinguish SAM (visual prompts only), Grounded SAM / SAM 2 (detector + SAM), and SAM 3 (native text prompts via Promptable Concept Segmentation)
Explain the SAM 3 architecture: shared backbone + image detector + memory-based video tracker + presence head + decoupled detector-tracker design
Use Hugging Face transformers SAM 3 integration for text-prompted detection, segmentation, and video tracking
Pick between SAM 3, Grounded SAM 2, YOLO-World, and SAM-MI based on latency, concept complexity, and deployment target

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.

Key terms in this lesson

Latency