Phase 6: Prompt Engineering & GenAI Feature Design · 40 min · Vision APIs · Voice/audio APIs · Image generation
The Problem
A PM on an insurance-claims product is excited about a new capability: users photograph vehicle damage, and a vision-enabled model estimates repair cost directly from the image, skipping the current flow where an adjuster manually reviews photos. It demos beautifully on a handful of clear, well-lit sample photos. In production, real users submit blurry photos, photos taken at odd angles, photos with poor lighting, and photos that only capture part of the damage — and the model still returns a confident-sounding dollar estimate every time, with no visible difference in confidence between a clear shot and an unusable one. The team didn't ship a worse version of the text-based flow; they shipped a feature with an entirely new failure surface (image quality, angle, occlusion, lighting) that the team hadn't evaluated for, because the excitement about the capability outran the analysis of its specific tradeoffs.
This is the core lesson of multimodal product design: each modality — vision, voice, image generation — has product patterns that work well, and a distinct set of failure modes and cost/latency/accuracy tradeoffs that don't map directly from what you already know about text.
Vision: product patterns and constraints
Vision-enabled models (processing images or video alongside text) unlock real product patterns: document/receipt extraction, defect or damage inspection (the claims example above), UI screenshot understanding, accessibility description generation. The tradeoffs that differ from text:
- Input quality is now a variable the product doesn't control, the way it doesn't fully control text quality either, but with a wider and harder-to-detect failure range — a blurry, poorly-lit, or badly-cropped image degrades output quality in a way the model may not signal, unlike a garbled text input which is often visibly nonsensical to a reviewer.
- Cost and latency scale with image resolution and count, not just prompt length — a feature analyzing multiple high-resolution images per request has a materially different cost profile than a single short text prompt, and this needs its own line in the cost projection table from Phase 4.
- Accuracy is highly sensitive to conditions the user controls (angle, lighting, framing) — which means the product often needs UX-level mitigation (capture guidance, a real-time quality check before submission) rather than trying to solve the whole problem at the model layer alone.
Voice/audio: product patterns and constraints
Voice products (speech-to-text, real-time voice agents, text-to-speech) unlock hands-free and accessibility-first interaction patterns, but introduce their own distinct tradeoffs:
- Latency is a UX-defining constraint in a way it rarely is for text. A chat interface tolerates a few seconds of "thinking" gracefully; a voice conversation with more than roughly a second of dead air starts to feel broken, because humans are extremely sensitive to unnatural pauses in spoken conversation. This changes the model/architecture tradeoff calculus directly — sometimes a smaller, faster model is the right product choice for a voice feature even if a larger model scores better on accuracy benchmarks.
- Transcription errors compound silently. A voice agent that mishears "not interested" as "interested" doesn't just make one mistake — everything downstream in that conversation inherits the error, often with no natural point where a user notices and corrects it the way they would notice and edit a visibly wrong typed message.
- Background noise, accents, and crosstalk are real-world conditions with no clean text analog, and accuracy claims from a vendor's demo environment often don't hold in the noisy conditions of real use (a call center floor, a moving car).
A modality-specific failure mode needs a modality-specific eval, not a generic one
The eval practices from Phase 5 -- golden datasets, category coverage, hallucination measurement -- all still apply to multimodal features, but the categories in the golden dataset need to reflect the modality's actual failure surface. A vision feature's golden set needs categories for image quality variation (blurry, poorly lit, occluded, wrong angle), not just content variation. A voice feature's golden set needs categories for background noise and accent variation, not just phrasing variation. Reusing a text-only eval structure for a multimodal feature will systematically miss the failure modes that are specific to the new modality -- exactly the gap in the claims-estimation example, where nobody built an eval category for 'blurry photo' before launch.
A voice-based customer service agent is evaluated using the same golden dataset categories as the team's existing text-chat product (happy path, edge case, known failure mode, adversarial), with no additional categories added. Three weeks after launch, the team discovers the agent performs poorly specifically on calls from users with strong regional accents, a failure mode nobody caught pre-launch. What was the root cause?
This is a direct evaluation-design gap, not an unavoidable surprise or a pure model-capability problem. Voice introduces failure modes (accent variation, background noise, transcription error) that have no equivalent category in a text-only golden dataset, and simply reusing the text product's category structure guarantees those modality-specific risks go untested before launch. The fix isn't necessarily a different model -- it's extending the golden dataset with voice-specific categories (a range of accents, background noise conditions, and transcription-error-prone phrasing) so this exact failure surface gets caught before real users do.
Image generation: product patterns and constraints
Image-generation features (creating new images from a prompt, or editing existing ones) show up in product design tools, marketing content generators, and personalization features. Distinct tradeoffs here:
- Output consistency across regenerations is often weaker than users expect — asking for "the same character in a different pose" frequently produces a visibly different character, a limitation with no text equivalent (regenerating text from the same prompt is far more likely to preserve intended meaning even when wording varies).
- Rights, likeness, and brand-safety risk is a product-level concern, not just a legal afterthought — an image-generation feature can produce output resembling a real person, a copyrighted style, or a competitor's trademark closely enough to be a real business risk, which needs a product-level mitigation (content filtering, usage restrictions) specified up front, not discovered after a user reports it.
- Cost per generation is usually much higher than a comparable text generation, which changes the economics of "let the user regenerate until they're happy" — an unlimited-regeneration UX pattern that's cheap for text can be materially expensive at scale for images.
A marketing content tool lets users generate unlimited logo variations with no rate limit, reasoning that "more regeneration options mean happier users." After launch, per-user image generation costs are far above the original cost projection, and the team also discovers several generated logos closely resemble existing trademarked brand marks. What does this reveal about the original design decision?
Both problems trace back to the same root cause: applying a mental model built for cheap, low-risk text regeneration to a modality with a meaningfully different cost structure and a meaningfully different risk profile. Cost per image generation is typically much higher than text, so 'unlimited regeneration' needed its own line in the cost projection table before launch, not an assumption carried over from a different modality. Separately, image generation carries a brand-safety and rights risk with no clean text analog, which needed an explicit product-level mitigation (content filtering, usage restrictions) specified up front. A cheaper model alone doesn't address the trademark risk, and blanket opposition to rate limits ignores that the limit here is solving a real, modality-specific problem.