Phase 3: Video Generation · 45 min · Python · fal.ai API · Runway SDK

Video Generation Modes

Image-to-video is the most common production pattern — you generate a perfect image first, then animate it.

Hiring signal: Knowing all video generation modes (text-to-video, image-to-video, video-to-video, first/last frame) and when to use each is fundamental to video generation engineering.

What you will learn

Implement text-to-video: generate video from text prompt only
Implement image-to-video: animate a static image — the most common production pattern
Understand video-to-video: transform existing video (style transfer, enhancement)
Use first-frame + last-frame: control start and end frames, model generates in-between

The Problem

A team uses text-to-video for everything. They write a prompt: "a cat sitting on a windowsill watching rain." The video comes out with a cat that looks different from frame to frame — wrong color, wrong pose, morphing face. They try again with a more detailed prompt. Same problem. They don't realize that text-to-video gives the model too much freedom — it decides what the cat looks like, and that decision drifts across frames.

The solution: generate a perfect image first with FLUX (where you have full control), then use image-to-video to animate it. The image anchors the visual identity, and the video model only handles motion. This is the most common production pattern.

What you'll build
Implement all four video generation modes: text-to-video, image-to-video, video-to-video, and first/last frame. Compare results and document when to use each mode.

Text-to-Video (T2V)

Generate video from a text prompt only.

# Text-to-video via fal.ai (Kling)
import requests

response = requests.post(
    "https://fal.run/fal-ai/kling-video/text-to-video",
    headers={"Authorization": f"Key {FAL_KEY}"},
    json={
        "prompt": "a cat sitting on a windowsill watching rain fall outside",
        "duration": "5",
        "aspect_ratio": "16:9",
    }
)

Pros: simplest, no input assets needed Cons: least control over visual identity, model decides everything When to use: concept exploration, when you don't have a specific visual in mind

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Image-to-Video (I2V), Video-to-Video (V2V), First Frame + Last Frame, Mode Comparison, Production I2V Pipeline, Key Takeaways, What's Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.