Phase 2: Tool Design & Function Calling · 50 min · Python · Pydantic · OpenAI SDK
The Problem
Two agents have a search tool. Agent A's tool is called search, takes a query string, and has the description "search for information." Agent B's tool is called search_product_catalog, takes query, category, and max_price, and has the description "Search the product catalog by name or description. Use this for questions about available products, pricing, or inventory. For order history, use get_order_history instead."
Agent B's tool selection accuracy is 94%. Agent A's is 61%. The difference is entirely in the schema design — no model change, no prompt change, no architecture change.
The anatomy of a great tool schema
A tool schema has four components, each with a job to do:
Name: Use a specific verb+noun pattern that is unambiguous even when surrounded by other tools. get_order_status not search. search_product_catalog not lookup. The model uses the name as a primary signal for whether a tool is relevant — generic names create ambiguity.
Description: Answer three questions for the model: (1) When should I call this tool — including situations where it might seem relevant but isn't? (2) What does this tool return? (3) What are the alternatives when this tool isn't the right one? A description that only says "gets order data" gives the model nothing to work with. A description that says "Returns real-time order status including shipping updates. For order history and past purchases, use get_order_history instead." gives the model a complete usage policy.
Parameters: Use the most specific type available. When a parameter has a fixed set of valid values, use an enum — the model will only produce valid values and will never hallucinate "in_transit" or "processing" when your system only accepts "pending", "shipped", "delivered", "cancelled". Include a description on every parameter, even obvious ones. For free-form parameters, include examples.
Required vs. optional: Mark a parameter as required only if the tool cannot function without it. A tool with too many required parameters creates a catch-22: the agent needs the tool to get information, but needs information to call the tool. Review each required parameter — if there's a sensible default, make it optional.
You're designing a tool that queries orders. Which parameter design is better for an "order_status" filter?
Enum type is the strongest constraint — the API and model both know exactly what values are valid, eliminating hallucinated inputs like "in_transit". Including "Omit to return all" tells the model when NOT to pass the parameter. Listing the values in the description is human-readable but doesn't constrain the model. Numeric codes (0=pending, 1=shipped…) are magic numbers, an anti-pattern that makes schemas unreadable.
Parameter design patterns
Enum parameters are the highest-precision tool in schema design. When a parameter can only take a fixed set of values — statuses, categories, sort orders, time ranges — use an enum. The model respects enum constraints with high reliability.
String parameters with examples work when values are free-form but have expected patterns. Include examples in the description: "Product name or description. Examples: 'wireless headphones', 'laptop under $1000', 'blue running shoes size 10'".
Optional parameters with explicit default descriptions let the model make an informed decision. Don't just mark a parameter as optional — tell the model what happens when it's omitted: "sort_by (optional): Sort order. Default: 'date_desc' (most recent first). Options: 'date_asc', 'relevance', 'price_asc'." The model can then decide whether the default is appropriate for the current task.
Nested objects reduce tool calling reliability. The model handles flat parameter lists more reliably than nested objects. Flatten where possible. If nesting is unavoidable, document the full structure with examples in the description.
Common schema anti-patterns
Audit your tool library against this list. Each anti-pattern has a measurable impact on selection accuracy:
| Anti-Pattern | Impact | Fix |
|---|
Generic name (search, get, process) | 20-30% accuracy drop from name ambiguity | Rename to verb+noun+context |
| Missing description | Model guesses from name alone | Write when-to-use and what-it-returns |
| Description explains implementation, not purpose | Model doesn't know when to call it | Rewrite from model's perspective |
| No parameter descriptions | Model guesses parameter values | Add description + examples to every param |
| String where enum works | Model hallucinates invalid values | Convert to enum with all valid values |
| Too many required parameters | Tool never called — model lacks all values | Review each: can it be optional with a default? |
| Too many optional parameters with no guidance | Model passes wrong values or omits needed ones | Add when-to-use notes to each optional param |
Your tool has 8 required parameters. The model is failing to call it because it often doesn't have all the information. What should you do?
A tool with too many required parameters creates a catch-22: the agent needs the tool to get information but needs information to call the tool. Review each parameter: if there's a sensible default (e.g., sort_by="date_desc", limit=10), make it optional with a documented default. The model can always override an optional parameter when it has a better value.
Build
Build it
The code artifact below implements a ToolSchema builder with Pydantic, an anti-pattern auditor, and a selection accuracy tester. Run the accuracy comparison to see the measurable difference between weak and strong schemas on the same 50-query test set.
"""
lesson_02_01_tool_schemas.py
Tool schema design with Pydantic.
Demonstrates: schema building, anti-pattern auditing, selection accuracy testing.
Run:
pip install openai pydantic
export OPENAI_API_KEY=...
python lesson_02_01_tool_schemas.py
"""
import json
import os
import re
from dataclasses import dataclass, field
from enum import Enum
from typing import Any, Optional
from openai import OpenAI
from pydantic import BaseModel, Field
# ---------------------------------------------------------------------------
# Pydantic models for tool parameter schemas
# ---------------------------------------------------------------------------
class OrderStatus(str, Enum):
PENDING = "pending"
SHIPPED = "shipped"
DELIVERED = "delivered"
CANCELLED = "cancelled"
class SortOrder(str, Enum):
DATE_DESC = "date_desc"
DATE_ASC = "date_asc"
RELEVANCE = "relevance"
PRICE_ASC = "price_asc"
PRICE_DESC = "price_desc"
class SearchProductsParams(BaseModel):
query: str = Field(
description="Product name or description to search for. Examples: 'wireless headphones', 'laptop under $1000', 'blue running shoes'."
)
category: Optional[str] = Field(
default=None,
description="Product category to filter by. Examples: 'electronics', 'clothing', 'home-garden'. Omit to search all categories.",
)
max_price: Optional[float] = Field(
default=None,
description="Maximum price in USD. Omit to return results at any price point.",
)
sort_by: SortOrder = Field(
default=SortOrder.RELEVANCE,
description="Sort order for results. Default: relevance. Use price_asc when the user wants the cheapest options.",
)
limit: int = Field(
default=10,
ge=1,
le=50,
description="Number of results to return. Default: 10. Use 5 for quick lookups, 50 for browsing.",
)
class GetOrderStatusParams(BaseModel):
order_id: str = Field(
description="The order ID to look up. Format: ORD-XXXXXXXX. For order history, use get_order_history instead."
)
class GetOrderHistoryParams(BaseModel):
customer_id: str = Field(
description="Customer ID to retrieve order history for."
)
status_filter: Optional[OrderStatus] = Field(
default=None,
description="Filter by order status. Omit to return all orders.",
)
limit: int = Field(
default=20,
description="Number of orders to return. Default: 20 (most recent).",
)
# ---------------------------------------------------------------------------
# Tool schema builder
# ---------------------------------------------------------------------------
def pydantic_to_openai_tool(
name: str,
description: str,
params_model: type[BaseModel],
) -> dict:
"""Convert a Pydantic model into an OpenAI tool definition."""
schema = params_model.model_json_schema()
# Remove title from top level (OpenAI doesn't need it)
schema.pop("title", None)
return {
"type": "function",
"function": {
"name": name,
"description": description,
"parameters": schema,
},
}
STRONG_TOOLS = [
pydantic_to_openai_tool(
name="search_product_catalog",
description=(
"Search the product catalog by name or description. Use this for questions about "
"available products, pricing, inventory, or product details. "
"For order status, use get_order_status. For order history, use get_order_history."
),
params_model=SearchProductsParams,
),
pydantic_to_openai_tool(
name="get_order_status",
description=(
"Returns real-time status of a specific order: current status, tracking number, "
"estimated delivery date. Requires the order ID. "
"For a list of past orders, use get_order_history instead."
),
params_model=GetOrderStatusParams,
),
pydantic_to_openai_tool(
name="get_order_history",
description=(
"Returns the list of past orders for a customer. Use when the user asks about "
"past purchases, order history, or wants to review previous orders. "
"For a single order's current status, use get_order_status."
),
params_model=GetOrderHistoryParams,
),
]
# Intentionally weak schemas — for comparison
WEAK_TOOLS = [
{
"type": "function",
"function": {
"name": "search",
"description": "search for information",
"parameters": {
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"],
},
},
},
{
"type": "function",
"function": {
"name": "get_order",
"description": "get order data",
"parameters": {
"type": "object",
"properties": {
"id": {"type": "string"},
"type": {"type": "string"},
},
"required": ["id", "type"],
},
},
},
]
# ---------------------------------------------------------------------------
# Schema anti-pattern auditor
# ---------------------------------------------------------------------------
@dataclass
class SchemaWarning:
tool_name: str
severity: str # "error" | "warning" | "info"
message: str
def audit_tool_schema(tool: dict) -> list[SchemaWarning]:
"""Check a tool definition for common schema anti-patterns."""
warnings: list[SchemaWarning] = []
fn = tool.get("function", {})
name = fn.get("name", "")
description = fn.get("description", "")
params = fn.get("parameters", {})
properties = params.get("properties", {})
# Check name quality
generic_names = {"search", "get", "fetch", "lookup", "query", "process", "handle", "do"}
if name.lower() in generic_names:
warnings.append(SchemaWarning(name, "error", f"Generic tool name '{name}'. Use a specific verb+noun like 'search_product_catalog'."))
if "_" not in name and name.lower() == name:
warnings.append(SchemaWarning(name, "warning", "Tool name has no noun context. Add a domain noun (e.g., 'get_order_status' not 'get')."))
# Check description quality
if not description:
warnings.append(SchemaWarning(name, "error", "Missing description. The model cannot select this tool appropriately."))
elif len(description) < 20:
warnings.append(SchemaWarning(name, "warning", f"Very short description ({len(description)} chars). Add when-to-use and what-it-returns."))
# Check parameter descriptions
for param_name, param_schema in properties.items():
if "description" not in param_schema:
warnings.append(SchemaWarning(name, "warning", f"Parameter '{param_name}' has no description. Add usage guidance and examples."))
# Check for strings that should be enums
if param_schema.get("type") == "string" and param_name in {"status", "type", "category", "sort", "sort_by", "order_by", "format"}:
if "enum" not in param_schema:
warnings.append(SchemaWarning(name, "info", f"Parameter '{param_name}' might benefit from an enum type if values are constrained."))
# Check for too many required parameters
required = params.get("required", [])
if len(required) > 5:
warnings.append(SchemaWarning(name, "warning", f"{len(required)} required parameters is high. Review which can be optional with sensible defaults."))
return warnings
# ---------------------------------------------------------------------------
# Selection accuracy tester
# ---------------------------------------------------------------------------
TEST_QUERIES = [
("Do you have wireless headphones under $200?", "search_product_catalog"),
("What's the status of my order ORD-12345678?", "get_order_status"),
("Show me my last 5 orders", "get_order_history"),
("I'm looking for running shoes in size 10", "search_product_catalog"),
("Where is my package from last week?", "get_order_status"),
("What products do you have in the electronics category?", "search_product_catalog"),
("Can you show me my cancelled orders?", "get_order_history"),
("Is the iPhone 15 in stock?", "search_product_catalog"),
("When will ORD-99887766 arrive?", "get_order_status"),
("I want to see everything I've bought in the past year", "get_order_history"),
]
def measure_accuracy(tools: list[dict], test_queries: list[tuple], client: OpenAI) -> dict:
"""Run test queries and measure tool selection accuracy."""
correct = 0
results = []
for query, expected_tool in test_queries:
try:
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a helpful e-commerce assistant. Use the available tools to help customers."},
{"role": "user", "content": query},
],
tools=tools,
tool_choice="auto",
)
tool_calls = response.choices[0].message.tool_calls
if tool_calls:
called_tool = tool_calls[0].function.name
else:
called_tool = "none"
is_correct = called_tool == expected_tool
if is_correct:
correct += 1
results.append({
"query": query,
"expected": expected_tool,
"called": called_tool,
"correct": is_correct,
})
except Exception as e:
results.append({"query": query, "expected": expected_tool, "called": "error", "correct": False, "error": str(e)})
accuracy = correct / len(test_queries) if test_queries else 0
return {"accuracy": accuracy, "correct": correct, "total": len(test_queries), "results": results}
# ---------------------------------------------------------------------------
# Main demo
# ---------------------------------------------------------------------------
if __name__ == "__main__":
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
print("=" * 60)
print("SCHEMA ANTI-PATTERN AUDIT")
print("=" * 60)
for tool_set_name, tool_set in [("WEAK TOOLS", WEAK_TOOLS), ("STRONG TOOLS", STRONG_TOOLS)]:
print(f"\n--- {tool_set_name} ---")
all_warnings = []
for tool in tool_set:
warnings = audit_tool_schema(tool)
all_warnings.extend(warnings)
if all_warnings:
for w in all_warnings:
icon = {"error": "✗", "warning": "△", "info": "ℹ"}.get(w.severity, "?")
print(f" {icon} [{w.tool_name}] {w.message}")
else:
print(" No issues found.")
print("\n" + "=" * 60)
print("SELECTION ACCURACY COMPARISON")
print("=" * 60)
print("\nTesting WEAK schemas...")
weak_results = measure_accuracy(WEAK_TOOLS, TEST_QUERIES, client)
print(f"Accuracy: {weak_results['accuracy']:.0%} ({weak_results['correct']}/{weak_results['total']})")
print("\nTesting STRONG schemas...")
strong_results = measure_accuracy(STRONG_TOOLS, TEST_QUERIES, client)
print(f"Accuracy: {strong_results['accuracy']:.0%} ({strong_results['correct']}/{strong_results['total']})")
print(f"\nImprovement: +{(strong_results['accuracy'] - weak_results['accuracy']):.0%}")
print("\n--- Per-query breakdown ---")
print(f"{'Query':<50} {'Weak':<12} {'Strong'}")
print("-" * 80)
for weak_r, strong_r in zip(weak_results["results"], strong_results["results"]):
query_short = weak_r["query"][:47] + "..." if len(weak_r["query"]) > 50 else weak_r["query"]
weak_icon = "✓" if weak_r["correct"] else "✗"
strong_icon = "✓" if strong_r["correct"] else "✗"
print(f"{query_short:<50} {weak_icon + ' ' + weak_r['called']:<12} {strong_icon} {strong_r['called']}")
print("\n--- Strong tool schemas (JSON) ---")
print(json.dumps(STRONG_TOOLS, indent=2))
Try tweaking the tool descriptions and parameter types below to see how a simple keyword-overlap "model" changes which tool it picks — a rough stand-in for how a real LLM weighs schema quality when selecting a tool.
# Simulate tool selection scored by how well a query matches
# each tool's name + description + enum values (a crude stand-in
# for an LLM reading the schema).
tools = {
"search": {
"description": "search for information",
"keywords": set("search for information".split()),
},
"search_product_catalog": {
"description": "Search the product catalog by name or description. Use for products, pricing, inventory.",
"keywords": set("search product catalog name description products pricing inventory".split()),
},
"get_order_status": {
"description": "Returns real-time order status, tracking number, estimated delivery.",
"keywords": set("order status tracking delivery real-time".split()),
},
"get_order_history": {
"description": "Returns list of past orders for a customer. Use for order history.",
"keywords": set("past orders history customer purchases".split()),
},
}
def select_tool(query):
query_words = set(query.lower().replace("?", "").split())
scores = {}
for name, tool in tools.items():
overlap = query_words & tool["keywords"]
scores[name] = len(overlap)
best = max(scores, key=scores.get)
return best, scores
queries = [
"What's the status of my order ORD-12345?",
"Show me my last 5 orders",
"Do you have wireless headphones under $200?",
]
for q in queries:
tool, scores = select_tool(q)
print(f"Query: {q}")
print(f" Selected: {tool} (scores: {scores})")