Phase 7: Voice Agent Frameworks · 55 min · Python · LiveKit Agents SDK · Deepgram SDK
The Concept
LiveKit Architecture
| Component | Role |
|---|
| LiveKit Server | WebRTC SFU (Selective Forwarding Unit) — routes audio |
| Agent Worker | Python process that connects to LiveKit room |
| Room | Audio/video session with participants |
| Participant | Agent or user in the room |
| Track | Audio stream from a participant |
LiveKit Agents Framework
from livekit.agents import Agent, AgentSession, JobContext
from livekit.plugins import openai, deepgram, cartesia
async def entrypoint(ctx: JobContext):
"""Agent entrypoint — called when a user joins the room."""
await ctx.connect()
session = AgentSession(
stt=deepgram.STT(model="nova-2"),
llm=openai.LLM(model="gpt-4o-mini"),
tts=cartesia.TTS(voice="agent-voice"),
)
await session.start(room=ctx.room)
class VoiceAgent(Agent):
def __init__(self):
super().__init__(instructions="""You are a helpful voice agent.
Keep responses short (1-3 sentences). Be conversational.""")
async def on_user_turn_completed(self, chat_ctx, new_message):
"""Called when user finishes speaking."""
# LiveKit handles VAD, endpointing, and barge-in automatically
pass
What's the key advantage of LiveKit's WebRTC-first approach over WebSocket-based frameworks?
Built-in echo cancellation, NAT traversal, and lower latency (~50ms vs ~100ms)
LiveKit vs Pipecat
| Factor | LiveKit | Pipecat |
|---|
| Transport | WebRTC (native) | WebSocket (default) |
| Echo cancellation | Built-in (WebRTC AEC) | Manual |
| NAT traversal | Built-in (ICE/STUN/TURN) | None |
| Latency | ~50ms | ~100ms |
| VAD/Endpointing | Built-in (Agents framework) | Manual or plugin |
| Barge-in | Built-in | CancelFrame |
| Browser support | Native WebRTC API | WebSocket |
| Telephony | Via LiveKit SIP | Via transport plugins |
| Best for | Browser/mobile agents | Server-to-server, telephony |
LiveKit Plugins
| Plugin | Service | Purpose |
|---|
livekit.plugins.openai | OpenAI | LLM (GPT-4o, GPT-4o-mini) |
livekit.plugins.deepgram | Deepgram | STT (Nova-2) |
livekit.plugins.cartesia | Cartesia | TTS (Sonic) |
livekit.plugins.elevenlabs | ElevenLabs | TTS |
livekit.plugins.silero | Silero | VAD |
livekit.plugins.rag | Custom | RAG integration |
Automatic VAD and Endpointing
LiveKit Agents handles VAD and endpointing automatically:
session = AgentSession(
stt=deepgram.STT(model="nova-2"),
llm=openai.LLM(model="gpt-4o-mini"),
tts=cartesia.TTS(voice="agent-voice"),
# VAD is built-in — no manual configuration needed
# Endpointing is automatic — detects when user finishes speaking
# Barge-in is automatic — stops TTS when user starts speaking
)
Custom Tools in LiveKit
from livekit.agents import function_tool
class VoiceAgent(Agent):
@function_tool
async def check_flight_status(self, flight_number: str) -> str:
"""Check the status of a flight."""
# This tool is automatically available to the LLM
status = await get_flight_status(flight_number)
return f"Flight {flight_number} is {status}"
@function_tool
async def book_flight(self, destination: str, date: str) -> str:
"""Book a flight to a destination on a specific date."""
booking = await create_booking(destination, date)
return f"Booked flight to {destination} on {date}. Confirmation: {booking.id}"
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Key Terms, Common Pitfalls, Interview Framing — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.