Phase 3: Voice, Video, Safety & Tutoring · 30 hours · Python (pipeline) · TypeScript (UI)

Capstone 12 — Video Understanding Pipeline (Scene, QA, Search)

Twelve Labs productized Marengo + Pegasus.

Hiring signal: Can build capstone 12 end to end

Introduction

VideoDB shipped the CRUD-for-video API. AI2's Molmo 2 published open VLM checkpoints. Gemini long-context handles hours of video natively. TimeLens-100K defined temporal grounding at scale. The 2026 pipeline is settled: scene segmentation, per-scene caption + embedding, transcript alignment, multi-vector index, and a query that answers with (start, end) timestamps plus frame previews. The capstone is ingesting 100 hours, hitting public benchmarks, and measuring hallucination on counting and action questions.

Phases exercised: P4 · P6 · P7 · P11 · P12 · P17

Type: Capstone Languages: Python (pipeline), TypeScript (UI) Prerequisites: Phase 4 (CV), Phase 6 (speech), Phase 7 (transformers), Phase 11 (LLM engineering), Phase 12 (multimodal), Phase 17 (infrastructure) Time: 30 hours

Objective

Learning objectives

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.