Phase 1: Advanced Prompt Engineering for Agents · 50 min · OpenAI SDK · Python · NumPy
The Problem
You defined a tool called search_orders. The model keeps calling it to answer questions about shipping policies, even though you have a separate get_policy tool for that.
You update the search_orders description to say "Use this for order-specific queries, not policy questions." The problem improves but doesn't disappear. A few days later you notice the model is now calling get_policy for order lookups when users phrase things like "what's the status of my recent order?"
The fundamental issue: descriptions tell the model what a tool IS. Examples show the model when to USE it. For ambiguous cases, examples are more reliable than descriptions.
Why agents need few-shot more than chatbots
For a chatbot, a wrong answer is annoying. For an agent with write access, calling the wrong tool can send the wrong email, delete the wrong record, or charge the wrong card. The cost of tool selection errors is orders of magnitude higher.
Why few-shot works for agents
When the model processes few-shot examples, it's not just memorizing — it's pattern-matching against the examples before generating its response. Concrete examples anchor the model to your specific vocabulary, scope, and edge cases in a way that descriptions alone cannot.
Compare:
Instruction only:
Use get_order_status for questions about specific customer orders.
Use get_shipping_policy for questions about shipping rules and timelines.
Instruction + example:
Use get_order_status for questions about specific customer orders.
Use get_shipping_policy for questions about shipping rules and timelines.
Example:
User: "What's your standard return window?"
→ Call: get_shipping_policy(topic="return_policy")
[NOT get_order_status — user hasn't mentioned a specific order]
The example makes concrete what "specific customer orders" means in practice — it needs an order ID or reference to a particular purchase.
Writing effective few-shot examples
A few-shot example for tool use must show the complete trace, not just input → output:
User: "Where is my order #12345?"
Thought: User is asking about a specific order by ID. Use get_order_status.
Do NOT use get_shipping_policy — this is about a specific order, not general policy.
Tool call: get_order_status(order_id="12345")
Tool result: {"status": "shipped", "carrier": "FedEx", "eta": "2026-07-02"}
Response: "Your order #12345 has shipped via FedEx and is expected to arrive July 2nd."
The critical elements:
- User message — shows the trigger pattern
- Thought — shows the reasoning (optional but helps with complex disambiguation)
- Tool call with arguments — shows exact function and parameters
- Tool result — shows what the model receives back
- Final response — shows how the model uses the result
The 2–5 example sweet spot
Research and practice converge on 2–5 high-quality examples. More doesn't mean better: each example adds tokens to every request, costs money, and can dilute the model's attention. Pick examples that cover the most common ambiguous cases, not every possible input.
Negative examples
Negative examples show what NOT to do and why. They're often more powerful than positive examples for preventing specific failure modes:
User: "How long does shipping take?"
Thought: User is asking about general shipping policy — how long it takes in general.
Do NOT call get_order_status — user hasn't mentioned a specific order.
Tool call: get_shipping_policy(topic="delivery_timeline")
Response: "Standard shipping takes 5–7 business days. Expedited is 2–3 days."
[NEGATIVE EXAMPLE]
User: "When will my package arrive?"
Thought: User is asking about a specific delivery, but has not provided an order ID.
I should NOT call get_order_status without an order ID — it will fail.
I should ASK for the order ID first.
Action: Ask for order ID before calling any tool.
Response: "I'd be happy to check that! Could you provide your order ID?"
Negative examples are especially valuable for:
- Preventing calls without required parameters
- Teaching when to ask for clarification instead of guessing
- Handling ambiguous inputs where two tools could plausibly be called
Your agent has tools: get_order_status, get_shipping_policy, escalate_to_human. It keeps calling get_order_status when users ask about return windows. What's the fastest fix?
A few-shot example that shows "return policy question → get_shipping_policy" directly anchors the correct behavior. Description changes help but take effect gradually and may introduce new ambiguities. A concrete example is the most reliable immediate fix for tool confusion.
Dynamic few-shot selection
Static few-shot examples use the same examples for every query. Dynamic few-shot selects the most relevant examples for each query by embedding similarity:
def select_examples(query: str, library: list[dict], top_k: int = 3) -> list[dict]:
query_embedding = embed(query)
example_embeddings = [embed(ex["user_message"]) for ex in library]
similarities = [
cosine_similarity(query_embedding, ex_emb)
for ex_emb in example_embeddings
]
top_indices = sorted(range(len(similarities)),
key=lambda i: similarities[i], reverse=True)[:top_k]
return [library[i] for i in top_indices]
Dynamic selection means a query about returns always gets the returns example, not the shipping timeline example — even if you have 50 examples in your library.
How many few-shot examples is typically optimal for tool selection in a production agent?
2–5 high-quality examples outperform both fewer (not enough signal) and more (noise, context length cost, diminishing returns). Each example costs tokens on every request. Choose examples that cover the most common ambiguous cases — not every possible input. With dynamic selection, you can maintain a larger library and only inject the most relevant examples per query.
Build
Build it
Here's a minimal, dependency-free version of dynamic example selection using simple word-overlap similarity instead of real embeddings — it shows the core idea: pick the example whose wording is closest to the query.
def word_overlap(a, b):
wa, wb = set(a.lower().split()), set(b.lower().split())
return len(wa & wb) / len(wa | wb) if wa or wb else 0
library = [
{"user_message": "Where is my order 12345?", "tool": "get_order_status"},
{"user_message": "What is your return policy?", "tool": "get_shipping_policy"},
{"user_message": "I am furious, nobody has helped me!", "tool": "escalate_to_human"},
]
queries = ["Track my order 98765", "How long until my package arrives?", "This is unacceptable, I want a manager"]
for query in queries:
scored = sorted(library, key=lambda ex: word_overlap(query, ex["user_message"]), reverse=True)
best = scored[0]
print(f"Query: {query!r}")
print(f" Closest example: {best['user_message']!r} -> {best['tool']}\n")
The code artifact implements a FewShotLibrary with static and dynamic example selection, plus a tool selection accuracy benchmark.
"""
lesson_01_02_few_shot.py
FewShotLibrary with:
- Static and dynamic (embedding-based) example selection
- Format validator for example completeness
- Tool selection accuracy benchmark: 30 queries with/without few-shot
- Before/after accuracy comparison by tool
Usage:
export OPENAI_API_KEY=sk-...
python lesson_01_02_few_shot.py
Requirements:
pip install openai numpy
"""
import json
import os
import numpy as np
from dataclasses import dataclass, field
from typing import Optional
import openai
@dataclass
class FewShotExample:
user_message: str
tool_called: str # Expected tool name
tool_args: dict
tool_result: str
final_response: str
thought: str = "" # Optional reasoning trace
is_negative: bool = False # If True: shows what NOT to do
@dataclass
class FewShotLibrary:
examples: list[FewShotExample] = field(default_factory=list)
_embeddings: Optional[np.ndarray] = field(default=None, init=False, repr=False)
def add(self, example: FewShotExample) -> "FewShotLibrary":
self.examples.append(example)
self._embeddings = None # Invalidate cache
return self
def validate(self, example: FewShotExample) -> list[str]:
issues = []
if not example.user_message:
issues.append("Missing user_message")
if not example.tool_called:
issues.append("Missing tool_called")
if not example.tool_args:
issues.append("Missing tool_args")
if not example.tool_result:
issues.append("Missing tool_result")
if not example.final_response:
issues.append("Missing final_response")
return issues
def format_for_prompt(self, examples: Optional[list[FewShotExample]] = None) -> str:
target = examples if examples is not None else self.examples
if not target:
return ""
lines = ["--- Tool Selection Examples ---"]
for ex in target:
if ex.is_negative:
lines.append(f"\n[NEGATIVE EXAMPLE — what NOT to do]")
lines.append(f"User: {ex.user_message}")
if ex.thought:
lines.append(f"Thought: {ex.thought}")
lines.append(f"Tool: {ex.tool_called}({json.dumps(ex.tool_args)})")
lines.append(f"Result: {ex.tool_result}")
lines.append(f"Response: {ex.final_response}")
lines.append("--- End Examples ---")
return "\n".join(lines)
def select_dynamic(
self, query: str, client: openai.OpenAI, top_k: int = 3
) -> list[FewShotExample]:
if not self.examples:
return []
messages = [ex.user_message for ex in self.examples]
all_texts = [query] + messages
resp = client.embeddings.create(
model="text-embedding-3-small",
input=all_texts,
)
vecs = np.array([item.embedding for item in resp.data], dtype=np.float32)
q_vec = vecs[0]
ex_vecs = vecs[1:]
norms = np.linalg.norm(ex_vecs, axis=1)
q_norm = np.linalg.norm(q_vec)
sims = ex_vecs @ q_vec / (norms * q_norm + 1e-10)
top_idx = np.argsort(sims)[::-1][:top_k]
return [self.examples[i] for i in top_idx]
TOOLS = [
{
"type": "function",
"function": {
"name": "get_order_status",
"description": (
"Look up the status of a specific customer order. "
"Use this when user mentions an order ID or asks about a specific purchase. "
"Requires an order_id. Do NOT use for general shipping policy questions."
),
"parameters": {
"type": "object",
"properties": {
"order_id": {"type": "string", "description": "The order ID"}
},
"required": ["order_id"],
},
},
},
{
"type": "function",
"function": {
"name": "get_shipping_policy",
"description": (
"Retrieve shipping policy information: timelines, rates, return windows, restrictions. "
"Use for general questions about how shipping works. "
"Do NOT use for questions about a specific order."
),
"parameters": {
"type": "object",
"properties": {
"topic": {
"type": "string",
"enum": ["delivery_timeline", "return_policy", "shipping_rates", "restrictions"],
"description": "Which policy topic to retrieve",
}
},
"required": ["topic"],
},
},
},
{
"type": "function",
"function": {
"name": "escalate_to_human",
"description": (
"Transfer the conversation to a human agent. "
"Use when the user is upset, the issue is too complex, "
"or a billing dispute needs human review."
),
"parameters": {
"type": "object",
"properties": {
"reason": {"type": "string", "description": "Reason for escalation"}
},
"required": ["reason"],
},
},
},
]
TEST_QUERIES = [
("get_order_status", "Where is my order #A1234?"),
("get_order_status", "Has order 99876 been shipped yet?"),
("get_shipping_policy","What's your return policy?"),
("get_shipping_policy","How long does standard shipping take?"),
("get_shipping_policy","Do you ship internationally?"),
("escalate_to_human", "This is outrageous, I want to speak to a manager now!"),
("get_order_status", "Track my recent purchase"),
("get_shipping_policy","What is the deadline for returning an item?"),
("escalate_to_human", "I've been waiting 3 weeks and nobody helps me, I'm furious"),
("get_shipping_policy","Can I return something bought 2 months ago?"),
]
def run_tool_accuracy_test(
client: openai.OpenAI,
test_cases: list[tuple[str, str]],
system_prompt: str,
model: str = "gpt-4o-mini",
) -> dict:
correct = 0
results = []
for expected_tool, query in test_cases:
resp = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": query},
],
tools=TOOLS,
tool_choice="auto",
max_tokens=100,
)
choice = resp.choices[0]
if choice.finish_reason == "tool_calls":
called = choice.message.tool_calls[0].function.name
else:
called = "no_tool"
is_correct = called == expected_tool
if is_correct:
correct += 1
results.append({
"query": query[:50],
"expected": expected_tool,
"called": called,
"correct": is_correct,
})
return {"accuracy": correct / len(test_cases), "results": results}
def main() -> None:
client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
# Build few-shot library
library = FewShotLibrary()
library.add(FewShotExample(
user_message="Where is my order #A1234?",
thought="User mentions a specific order ID — use get_order_status.",
tool_called="get_order_status",
tool_args={"order_id": "A1234"},
tool_result='{"status": "shipped", "carrier": "FedEx", "eta": "July 2"}',
final_response="Your order #A1234 has shipped via FedEx, arriving July 2.",
))
library.add(FewShotExample(
user_message="What's your return policy?",
thought="General policy question — no specific order. Use get_shipping_policy.",
tool_called="get_shipping_policy",
tool_args={"topic": "return_policy"},
tool_result='{"window": "30 days", "condition": "unused, original packaging"}',
final_response="You can return unused items in original packaging within 30 days.",
))
library.add(FewShotExample(
user_message="When will my package arrive?",
thought="User asks about delivery but provided no order ID. Ask for it first.",
tool_called="get_order_status",
tool_args={"order_id": "NEEDED"},
tool_result="",
final_response="I can look that up! Could you share your order ID?",
is_negative=True,
))
library.add(FewShotExample(
user_message="I'm furious, nobody has helped me for days!",
thought="User is upset — escalate immediately.",
tool_called="escalate_to_human",
tool_args={"reason": "Customer expressing frustration"},
tool_result='{"status": "escalated", "eta": "2 minutes"}',
final_response="I'm connecting you with a human agent right now.",
))
system_base = (
"You are a customer support agent for Acme Corp. "
"Use the provided tools to help customers with orders and shipping questions."
)
system_with_examples = system_base + "\n\n" + library.format_for_prompt()
print("FEW-SHOT TOOL SELECTION BENCHMARK")
print("=" * 70)
print(f"Test set: {len(TEST_QUERIES)} queries across 3 tools\n")
print("Running WITHOUT few-shot examples...")
baseline = run_tool_accuracy_test(client, TEST_QUERIES, system_base)
print("Running WITH few-shot examples...")
with_fewshot = run_tool_accuracy_test(client, TEST_QUERIES, system_with_examples)
print(f"\nBaseline accuracy: {baseline['accuracy']:.0%}")
print(f"With few-shot: {with_fewshot['accuracy']:.0%}")
improvement = with_fewshot['accuracy'] - baseline['accuracy']
print(f"Improvement: +{improvement:.0%}\n")
print(f"\n{'Query':<50} {'Expected':<22} {'Baseline':<22} {'Few-Shot'}")
print("─" * 100)
for b, f in zip(baseline["results"], with_fewshot["results"]):
b_icon = "✓" if b["correct"] else "✗"
f_icon = "✓" if f["correct"] else "✗"
print(f"{b['query']:<50} {b['expected']:<22} {b_icon} {b['called']:<20} {f_icon} {f['called']}")
if __name__ == "__main__":
main()