Phase 6: Prompt Engineering & GenAI Feature Design · 40 min · Vision APIs · Voice/audio APIs · Image generation

Multimodal Product Design

Adding a modality doesn't just add a capability. It adds a new set of ways the product can be confidently wrong.

Hiring signal: As AI products expand beyond text, PMs are expected to reason about modality-specific tradeoffs without a technical translator in the room. An interviewer asking 'would you use vision or a form for this' is testing whether a candidate defaults to the most exciting modality or actually weighs latency, cost, and accuracy tradeoffs specific to that input type -- the same judgment this course has built for text-based features, now applied to image, voice, and generated media.

What you will learn

Identify common product patterns for vision, voice, and image-generation features
Compare latency, cost, and accuracy tradeoffs across modalities for a given use case
Recognize modality-specific failure modes that don't have a direct text-based analog
Decide when a non-text modality is genuinely the right interface versus an unnecessary complexity add

The Problem

A PM on an insurance-claims product is excited about a new capability: users photograph vehicle damage, and a vision-enabled model estimates repair cost directly from the image, skipping the current flow where an adjuster manually reviews photos. It demos beautifully on a handful of clear, well-lit sample photos. In production, real users submit blurry photos, photos taken at odd angles, photos with poor lighting, and photos that only capture part of the damage — and the model still returns a confident-sounding dollar estimate every time, with no visible difference in confidence between a clear shot and an unusable one. The team didn't ship a worse version of the text-based flow; they shipped a feature with an entirely new failure surface (image quality, angle, occlusion, lighting) that the team hadn't evaluated for, because the excitement about the capability outran the analysis of its specific tradeoffs.

This is the core lesson of multimodal product design: each modality — vision, voice, image generation — has product patterns that work well, and a distinct set of failure modes and cost/latency/accuracy tradeoffs that don't map directly from what you already know about text.

Vision: Product Patterns and Constraints

Vision-enabled models (processing images or video alongside text) unlock real product patterns: document/receipt extraction, defect or damage inspection (the claims example above), UI screenshot understanding, accessibility description generation. The tradeoffs that differ from text:

Input quality is now a variable the product doesn't control, the way it doesn't fully control text quality either, but with a wider and harder-to-detect failure range — a blurry, poorly-lit, or badly-cropped image degrades output quality in a way the model may not signal, unlike a garbled text input which is often visibly nonsensical to a reviewer.
Cost and latency scale with image resolution and count, not just prompt length — a feature analyzing multiple high-resolution images per request has a materially different cost profile than a single short text prompt, and this needs its own line in the cost projection table from Phase 4.
Accuracy is highly sensitive to conditions the user controls (angle, lighting, framing) — which means the product often needs UX-level mitigation (capture guidance, a real-time quality check before submission) rather than trying to solve the whole problem at the model layer alone.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Voice/Audio: Product Patterns and Constraints, Image Generation: Product Patterns and Constraints, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.

Key terms in this lesson

Latency