Phase 4: LLM Orchestration for Voice · 55 min · Python · OpenAI SDK · Anthropic SDK

Tool/Function Calling During Voice Conversations

Tools must return in 2-3 seconds or the conversation feels broken. Latency-aware tool selection is non-negotiable.

Hiring signal: Tool calling within voice latency budgets is a core differentiator tested in voice AI coding rounds.

What you will learn

Implement tool/function calling during voice conversations within latency budgets
Select latency-aware tools: fast tools for real-time, async tools for post-call
Handle tool failures gracefully in spoken conversation
Format tool results for spoken output: concise, natural language summaries

The Problem

A voice agent that only talks is useless. Callers want action: "Check my balance", "Book me a flight", "Cancel my subscription". The LLM needs to call external APIs (tools) during the conversation, wait for results, and respond naturally — all within the latency budget.

The Concept

How Tool Calling Works

Tool Definition

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "check_balance",
            "description": "Check a customer's account balance",
            "parameters": {
                "type": "object",
                "properties": {
                    "user_id": {"type": "string", "description": "Customer ID"}
                },
                "required": ["user_id"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "book_flight",
            "description": "Search and book flights",
            "parameters": {
                "type": "object",
                "properties": {
                    "destination": {"type": "string"},
                    "date": {"type": "string", "description": "YYYY-MM-DD"},
                    "time_preference": {"type": "string", "enum": ["morning", "afternoon", "evening"]}
                },
                "required": ["destination", "date"]
            }
        }
    }
]

While the LLM waits for a tool API to respond (e.g., checking balance takes 500ms), what should the voice agent do?

Hang up and call back

Tool Calling Flow with Streaming

async def handle_tool_call(messages, user_input):
    """Handle LLM response with potential tool calls."""
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        tools=TOOLS,
        stream=True,
        max_tokens=150,
    )

    has_tool_call = False
    text_buffer = ""

    async for chunk in response:
        delta = chunk.choices[0].delta

        # Stream text to TTS
        if delta.content:
            text_buffer += delta.content
            yield {"type": "text", "content": delta.content}

        # Check for tool call
        if delta.tool_calls:
            has_tool_call = True
            tool_call = delta.tool_calls[0]
            yield {"type": "tool_call", "name": tool_call.function.name,
                   "args": tool_call.function.arguments}

    if has_tool_call:
        # Execute tool and get result
        result = await execute_tool(tool_call)
        # Send result back to LLM for final response
        yield {"type": "tool_result", "result": result}

Tool Execution

async def execute_tool(tool_name, args):
    """Execute a tool call and return the result."""
    if tool_name == "check_balance":
        return await check_balance(**args)
    elif tool_name == "book_flight":
        return await book_flight(**args)
    elif tool_name == "transfer_call":
        return await transfer_call(**args)

async def check_balance(user_id):
    """Check customer balance via API."""
    # Real implementation would call your API
    response = await httpx.get(f"https://api.company.com/balance/{user_id}")
    return response.json()  # {"balance": 1247.89}

Latency Impact of Tool Calls

Phase	Time	Caller Hears
ASR	150ms	(silence)
LLM TTFT	150ms	(silence)
Filler phrase TTS	120ms	"Let me check..."
Tool API call	200-500ms	(filler phrase playing)
LLM final response	150ms	(silence)
TTS final response	120ms	"Your balance is..."
Total	890-1190ms

The filler phrase masks the tool API latency. Without it, the caller would experience 500ms+ of silence.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.