Phase 3: Video Generation · 45 min · Python · fal.ai API · Runway SDK

Video Generation Modes

Image-to-video is the most common production pattern — you generate a perfect image first, then animate it.

Real-world application: Knowing all video generation modes (text-to-video, image-to-video, video-to-video, first/last frame) and when to use each is fundamental to video generation engineering.

What you will learn

Implement text-to-video: generate video from text prompt only
Implement image-to-video: animate a static image — the most common production pattern
Understand video-to-video: transform existing video (style transfer, enhancement)
Use first-frame + last-frame: control start and end frames, model generates in-between

Objective

Create motion evidence that can be reviewed frame by frame, not only as a successful API response. In the paired studio lab, build a pipeline that: (1) generates an image with FLUX, (2) animates it into a 5-second video using Kling or Runway image-to-video API. Demonstrate the full text → image → video flow and track costs at each stage.

The Problem

A team uses text-to-video for everything. They write a prompt: "a cat sitting on a windowsill watching rain." The video comes out with a cat that looks different from frame to frame — wrong color, wrong pose, morphing face. They try again with a more detailed prompt. Same problem. They don't realize that text-to-video gives the model too much freedom — it decides what the cat looks like, and that decision drifts across frames.

The solution: generate a perfect image first with FLUX (where you have full control), then use image-to-video to animate it. The image anchors the visual identity, and the video model only handles motion. This is the most common production pattern.

What you'll build
Implement all four video generation modes: text-to-video, image-to-video, video-to-video, and first/last frame. Compare results and document when to use each mode.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.