Phase 4: LLM Orchestration for Voice · 55 min · Python · OpenAI SDK · Anthropic SDK
The Concept
How Tool Calling Works
Tool Definition
TOOLS = [
{
"type": "function",
"function": {
"name": "check_balance",
"description": "Check a customer's account balance",
"parameters": {
"type": "object",
"properties": {
"user_id": {"type": "string", "description": "Customer ID"}
},
"required": ["user_id"]
}
}
},
{
"type": "function",
"function": {
"name": "book_flight",
"description": "Search and book flights",
"parameters": {
"type": "object",
"properties": {
"destination": {"type": "string"},
"date": {"type": "string", "description": "YYYY-MM-DD"},
"time_preference": {"type": "string", "enum": ["morning", "afternoon", "evening"]}
},
"required": ["destination", "date"]
}
}
}
]
While the LLM waits for a tool API to respond (e.g., checking balance takes 500ms), what should the voice agent do?
Dead air is the enemy of voice agents. The pattern is: (1) LLM recognizes intent to use a tool, (2) LLM generates a filler phrase ("Let me check that for you" / "One moment please") and streams it to TTS, (3) LLM calls the tool API, (4) Tool returns data (may take 200-500ms), (5) LLM generates final response with the data. The filler phrase covers the tool latency — the caller hears "Let me check that for you" while the API is being called, then hears the result. This is why the system prompt should include: "If you need to check something, call the tool and say 'Let me check that for you.'"
Tool Calling Flow with Streaming
async def handle_tool_call(messages, user_input):
"""Handle LLM response with potential tool calls."""
response = await client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=TOOLS,
stream=True,
max_tokens=150,
)
has_tool_call = False
text_buffer = ""
async for chunk in response:
delta = chunk.choices[0].delta
# Stream text to TTS
if delta.content:
text_buffer += delta.content
yield {"type": "text", "content": delta.content}
# Check for tool call
if delta.tool_calls:
has_tool_call = True
tool_call = delta.tool_calls[0]
yield {"type": "tool_call", "name": tool_call.function.name,
"args": tool_call.function.arguments}
if has_tool_call:
# Execute tool and get result
result = await execute_tool(tool_call)
# Send result back to LLM for final response
yield {"type": "tool_result", "result": result}
Tool Execution
async def execute_tool(tool_name, args):
"""Execute a tool call and return the result."""
if tool_name == "check_balance":
return await check_balance(**args)
elif tool_name == "book_flight":
return await book_flight(**args)
elif tool_name == "transfer_call":
return await transfer_call(**args)
async def check_balance(user_id):
"""Check customer balance via API."""
# Real implementation would call your API
response = await httpx.get(f"https://api.company.com/balance/{user_id}")
return response.json() # {"balance": 1247.89}
Latency Impact of Tool Calls
| Phase | Time | Caller Hears |
|---|
| ASR | 150ms | (silence) |
| LLM TTFT | 150ms | (silence) |
| Filler phrase TTS | 120ms | "Let me check..." |
| Tool API call | 200-500ms | (filler phrase playing) |
| LLM final response | 150ms | (silence) |
| TTS final response | 120ms | "Your balance is..." |
| Total | 890-1190ms | |
The filler phrase masks the tool API latency. Without it, the caller would experience 500ms+ of silence.