Phase 9: Advanced Topics & Capstone · 45 min · Python · Google Gemini SDK · OpenAI SDK
The Concept
Multimodal Architecture
| Input Mode | Example | Use Case |
|---|
| Voice only | "Book a flight to Tokyo" | Standard voice agent |
| Voice + image | "What's wrong with this error?" + screenshot | Tech support |
| Voice + video | "Is this rash concerning?" + camera | Telehealth |
| Voice + document | "Summarize this contract" + PDF upload | Legal/finance |
| Voice + screen share | "Walk me through this" + screen | Guided support |
What's the key challenge in building a multimodal voice agent compared to a voice-only agent?
The key challenge is input synchronization and fusion. A voice-only agent processes one stream (audio). A multimodal agent processes two+ streams at different latencies: ASR returns text in ~150ms, vision model returns image description in ~200ms. The agent must: (1) Correlate inputs — "this screenshot" refers to the image shared 2 seconds ago, not the current one. (2) Decide modality — should it ask "can you share a screenshot?" or "can you describe it?" based on the problem. (3) Handle missing modalities — caller says "look at this" but hasn't shared an image yet. (4) Manage bandwidth — video streaming uses 10x more bandwidth than audio. (5) Fusion strategy — does the LLM process both in one call (GPT-4o multimodal) or separately (ASR + vision API + LLM)? GPT-4o's native multimodal support is solving this, but latency and cost increase with each modality.
Multimodal LLM Integration
class MultimodalAgent:
"""Voice agent with vision capabilities.
asr: anything with `async transcribe(audio) -> str`
llm: a multimodal chat client with `async chat(messages=...) -> str`
"""
def __init__(self, asr, llm):
self.asr = asr
self.llm = llm
self.conversation = []
self.shared_images = []
async def handle_voice_with_image(self, audio_input, image_input=None):
"""Handle voice input with optional image."""
# Transcribe audio
transcript = await self.asr.transcribe(audio_input)
# If image provided, add to context
if image_input:
self.shared_images.append({
"image": image_input,
"timestamp": time.time(),
"context": transcript, # What was said when image was shared
})
# Build multimodal message
message = {"role": "user", "content": []}
message["content"].append({"type": "text", "text": transcript})
if image_input:
message["content"].append({
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{image_input}"}
})
# Generate response with multimodal LLM
response = await self.llm.chat(messages=[*self.conversation, message])
self.conversation += [message, {"role": "assistant", "content": response}]
return response
Screen Sharing for Support
class ScreenShareSupport:
"""Voice agent with screen sharing for tech support."""
def __init__(self, asr, llm):
self.asr = asr
self.llm = llm
async def handle_support_call(self, audio, screen_frame=None):
"""Handle a support call with optional screen sharing."""
transcript = await self.asr.transcribe(audio)
# Detect if user is describing a visual problem
visual_keywords = ["screen", "error", "display", "button", "menu", "see"]
needs_visual = any(kw in transcript.lower() for kw in visual_keywords)
if needs_visual and not screen_frame:
# Ask for screen share
return "I think I need to see your screen. Can you share it?"
if screen_frame:
# The multimodal model reads the screenshot itself; no separate
# vision pass is needed.
response = await self.llm.chat(
messages=[{
"role": "user",
"content": [
{"type": "text", "text": transcript},
{"type": "image_url", "image_url": {"url": screen_frame}},
]
}]
)
return response
return await self.llm.chat(messages=[{"role": "user", "content": transcript}])
Telehealth: Voice + Camera
A camera makes a health agent more useful and more dangerous. Software that diagnoses a condition or recommends treatment is regulated as a medical device (by the FDA in the US, under the MDR in the EU), and a model that confidently misreads a rash in poor light can do real harm. So the agent below triages and routes: it describes what it sees, asks about red flags, and tells the caller who should look at it and how soon. It never diagnoses.
TRIAGE_PROMPT = (
"You are a telehealth intake assistant, not a clinician. Describe what the patient "
"shows and says, ask about red flags (fever, spreading redness, trouble breathing, "
"severe pain), and recommend a level of care: emergency services now, same-day "
"clinician, or routine appointment. Never diagnose, name a condition as certain, "
"or recommend medication. If anything suggests an emergency, say so first."
)
class TelehealthAgent:
"""Voice agent with camera for telehealth intake and triage."""
def __init__(self, llm):
self.llm = llm
async def assess_condition(self, voice_description, camera_frame=None):
"""Assess patient condition with voice + visual."""
# Voice-only assessment
if not camera_frame:
return await self.llm.chat(
messages=[{
"role": "system",
"content": TRIAGE_PROMPT,
}, {
"role": "user",
"content": voice_description,
}]
)
# Voice + visual assessment
return await self.llm.chat(
messages=[{
"role": "system",
"content": TRIAGE_PROMPT + " You can also see the patient via camera.",
}, {
"role": "user",
"content": [
{"type": "text", "text": voice_description},
{"type": "image_url", "image_url": {"url": camera_frame}},
]
}]
)
Modality Decision Engine
class ModalityDecisionEngine:
"""Decides when to request visual input."""
VISUAL_TRIGGERS = [
"error", "screen", "display", "button", "menu", "see this",
"look at", "show you", "picture", "document", "form",
"rash", "wound", "injury", "diagram", "chart",
]
def should_request_visual(self, transcript):
"""Determine if visual input would help."""
transcript_lower = transcript.lower()
return any(trigger in transcript_lower for trigger in self.VISUAL_TRIGGERS)
def recommend_modality(self, transcript, has_image=False):
"""Recommend the best modality for this interaction."""
if self.should_request_visual(transcript) and not has_image:
return "request_image"
elif has_image:
return "multimodal"
else:
return "voice_only"
Native realtime multimodal APIs
Everything above is a pipeline: speech-to-text, then a multimodal model, then text-to-speech. Two hosted APIs collapse that into one streaming session:
- Gemini Live API keeps a bidirectional WebSocket open. You stream microphone audio and camera or screen frames (around one frame per second is typical) and receive spoken audio back, with interruption handled by the server.
- OpenAI Realtime API does the same for audio over WebSocket or WebRTC, and accepts images as input alongside it.
The trade is control for latency. A native session skips two network hops and keeps tone of voice, but you lose the transcript-level hooks the pipeline gives you: the modality decision engine, keyword triggers and per-stage latency budgets. The lab lets you build either; the concepts in this lesson apply to both.
Latency Impact
| Mode | Components | Latency |
|---|
| Voice only | ASR + LLM + TTS | ~500ms |
| Voice + image | ASR + Vision + LLM + TTS | ~700ms |
| Voice + video stream | ASR + Vision (per frame) + LLM + TTS | ~800ms |
| Screen share | ASR + Screen capture + Vision + LLM + TTS | ~750ms |