Phase 9: Advanced Topics & Capstone · 45 min · Python · Google Gemini SDK · OpenAI SDK
The Concept
Multimodal Architecture
| Input Mode | Example | Use Case |
|---|
| Voice only | "Book a flight to Tokyo" | Standard voice agent |
| Voice + image | "What's wrong with this error?" + screenshot | Tech support |
| Voice + video | "Is this rash concerning?" + camera | Telehealth |
| Voice + document | "Summarize this contract" + PDF upload | Legal/finance |
| Voice + screen share | "Walk me through this" + screen | Guided support |
What's the key challenge in building a multimodal voice agent compared to a voice-only agent?
Synchronizing audio and image inputs — the agent must correlate what it hears with what it sees, handle different processing latencies (ASR ~150ms vs vision ~200ms), and decide when to ask for visual input vs relying on voice alone
Multimodal LLM Integration
class MultimodalAgent:
"""Voice agent with vision capabilities."""
def __init__(self):
self.llm = OpenAIClient(model="gpt-4o") # Multimodal model
self.conversation = []
self.shared_images = []
async def handle_voice_with_image(self, audio_input, image_input=None):
"""Handle voice input with optional image."""
# Transcribe audio
transcript = await self.asr.transcribe(audio_input)
# If image provided, add to context
if image_input:
self.shared_images.append({
"image": image_input,
"timestamp": time.time(),
"context": transcript, # What was said when image was shared
})
# Build multimodal message
message = {"role": "user", "content": []}
message["content"].append({"type": "text", "text": transcript})
if image_input:
message["content"].append({
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{image_input}"}
})
# Generate response with multimodal LLM
response = await self.llm.chat(messages=[*self.conversation, message])
return response
Screen Sharing for Support
class ScreenShareSupport:
"""Voice agent with screen sharing for tech support."""
async def handle_support_call(self, audio, screen_frame=None):
"""Handle a support call with optional screen sharing."""
transcript = await self.asr.transcribe(audio)
# Detect if user is describing a visual problem
visual_keywords = ["screen", "error", "display", "button", "menu", "see"]
needs_visual = any(kw in transcript.lower() for kw in visual_keywords)
if needs_visual and not screen_frame:
# Ask for screen share
return "I think I need to see your screen. Can you share it?"
if screen_frame:
# Analyze screen with vision
screen_analysis = await self.vision.analyze(screen_frame)
response = await self.llm.chat(
messages=[{
"role": "user",
"content": [
{"type": "text", "text": transcript},
{"type": "image_url", "image_url": {"url": screen_frame}},
]
}]
)
return response
Telehealth: Voice + Camera
class TelehealthAgent:
"""Voice agent with camera for telehealth."""
async def assess_condition(self, voice_description, camera_frame=None):
"""Assess patient condition with voice + visual."""
# Voice-only assessment
if not camera_frame:
return await self.llm.chat(
messages=[{
"role": "system",
"content": "You are a healthcare assistant. Based on the description, ask clarifying questions."
}, {
"role": "user",
"content": voice_description,
}]
)
# Voice + visual assessment
return await self.llm.chat(
messages=[{
"role": "system",
"content": "You are a healthcare assistant. You can see the patient via camera and hear their description. Provide assessment."
}, {
"role": "user",
"content": [
{"type": "text", "text": voice_description},
{"type": "image_url", "image_url": {"url": camera_frame}},
]
}]
)
Modality Decision Engine
class ModalityDecisionEngine:
"""Decides when to request visual input."""
VISUAL_TRIGGERS = [
"error", "screen", "display", "button", "menu", "see this",
"look at", "show you", "picture", "document", "form",
"rash", "wound", "injury", "diagram", "chart",
]
def should_request_visual(self, transcript):
"""Determine if visual input would help."""
transcript_lower = transcript.lower()
return any(trigger in transcript_lower for trigger in self.VISUAL_TRIGGERS)
def recommend_modality(self, transcript, has_image=False):
"""Recommend the best modality for this interaction."""
if self.should_request_visual(transcript) and not has_image:
return "request_image"
elif has_image:
return "multimodal"
else:
return "voice_only"
Latency Impact
| Mode | Components | Latency |
|---|
| Voice only | ASR + LLM + TTS | ~500ms |
| Voice + image | ASR + Vision + LLM + TTS | ~700ms |
| Voice + video stream | ASR + Vision (per frame) + LLM + TTS | ~800ms |
| Screen share | ASR + Screen capture + Vision + LLM + TTS | ~750ms |