Phase 8: Production Agent Infrastructure · 90 min · Python · Anthropic SDK · HuggingFace TRL
The Problem
Your general-purpose model calls the wrong tool 15% of the time. It misformats tool arguments 8% of the time. It confuses process_refund and escalate_to_human for ambiguous queries. These error rates compound: in a 5-step agent workflow, a 15% per-step error rate means you complete correctly only 44% of the time.
The fix is not a better prompt. You've already tried that. The fix is fine-tuning: teaching a smaller model to deeply know your specific tools, argument schemas, and the mapping from user intents to correct tool calls. After fine-tuning on 500 examples, tool selection accuracy goes from 85% to 97%+. Schema adherence goes from 60% to 98%+.
And the fine-tuned 8B model costs 1/8 as much to run as the general 70B model you were relying on. This is the production case for fine-tuning: higher accuracy, lower cost, for stable and well-defined tool sets.
When to fine-tune vs. prompt engineer
Fine-tuning makes sense when: your tool set is stable (not changing weekly), you need consistent accuracy across high volumes (>10K tool calls/day), and a general model isn't reaching your accuracy target with few-shot prompting. Don't fine-tune if you're still iterating on your tools — every schema change requires regenerating data and retraining. Prompting is cheaper to iterate; fine-tuning is cheaper to operate at scale.
Why fine-tune for tool use
General models learn tool use from the tool schemas and few-shot examples in the prompt. This works — to a point. The model has never seen your specific tools before the prompt, and it's generalizing from a small number of examples.
Fine-tuned models have it baked in. Benefits:
- Higher accuracy: fewer wrong tool selections, fewer malformed arguments. The model has seen thousands of examples of correct tool calls and doesn't need to generalize from a few.
- Smaller prompt: you need fewer examples in the prompt (smaller system prompt). The fine-tuned model already knows the patterns.
- Consistency: the fine-tuned model applies tools more consistently across diverse user inputs, including edge cases, multilingual queries, and unusual phrasings.
- Cost efficiency: a fine-tuned Llama 3.2 3B model running on a $0.50/hour A10G GPU may match a general Claude 3.5 Sonnet on your specific tool calling task at 1/100th the per-call cost at volume.
A fine-tuned 8B model with 5,000 tool call examples vs. a general 70B model with 10 few-shot examples in the prompt. Which is likely better at tool calling?
Explanation: This is the core fine-tuning vs. prompting tradeoff. Fine-tuning is expensive upfront (training cost, data generation) but produces a model that deeply knows your specific tools. Few-shot prompting is flexible but relies on the model generalizing from a small context window. For production systems with stable, well-defined tool sets processing high volumes, a fine-tuned small model consistently outperforms general large models on tool accuracy — and costs a fraction as much per call.
Synthetic dataset generation
You need 500–5000 examples per tool for effective fine-tuning. Generating these manually is impractical. Use a teacher model (Claude Sonnet) to generate examples automatically.
SFT (Supervised Fine-Tuning) examples:
# Format: user message -> correct tool call
{
"messages": [{"role": "user", "content": "I want a refund for my damaged order ORD-12345678"}],
"assistant": {"tool": "process_refund", "arguments": {"order_id": "ORD-12345678", "reason": "defective"}},
"tools": list(tool_schemas)
}
The generation pipeline using Distilabel's SelfInstruct pattern:
- Provide seed queries (5–10 per tool)
- Distilabel generates diverse paraphrases and variations using a seed model
- Claude Sonnet generates correct tool calls for each variation (teacher labeling)
- Filter out low-confidence examples
Cover: all tools, all argument combinations, edge cases (missing info), multilingual, different user phrasings ("what's the status", "track my package", "where is it", "I haven't received anything").
DPO (Direct Preference Optimization) pairs:
Each DPO pair has a correct call (chosen) and an incorrect call (rejected) for the same query. The model learns to prefer the chosen response.
Three rejection types to generate:
wrong_tool: correct query but the wrong tool was selectedwrong_argument_type: right tool but a required string field gets a number valuemissing_required: right tool but a required field is omitted
# DPO pair format
{
"messages": [{"role": "user", "content": "Check my order ORD-12345678"}],
"chosen": {"tool": "get_order_status", "arguments": {"order_id": "ORD-12345678"}},
"rejected": {"tool": "search_products", "arguments": {}}, # wrong_tool rejection
"tools": list(tool_schemas)
}
DPO (Direct Preference Optimization) pairs for tool call fine-tuning consist of:
Explanation: DPO trains on preferences: "Given this query and these tools, this tool call is BETTER (chosen) than this tool call (rejected)." The model adjusts weights to increase the likelihood of chosen responses and decrease rejected ones. For tool calling, chosen = correct tool + correct arguments; rejected = wrong tool, wrong argument types, or missing required fields. This directly addresses the failure modes you observe in production.
Training and evaluation pipeline
Training with HuggingFace TRL:
from trl import SFTTrainer, SFTConfig
from transformers import AutoModelForCausalLM
import peft # LoRA for efficient fine-tuning
# 1. Load base model with 4-bit quantization (fits on 24GB GPU)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B-Instruct", load_in_4bit=True)
# 2. SFT training (3 epochs, 500 examples)
trainer = SFTTrainer(model=model, train_dataset=sft_dataset,
args=SFTConfig(num_train_epochs=3, per_device_train_batch_size=4, learning_rate=2e-4))
trainer.train()
# 3. Optional: DPO training on top of SFT model
from trl import DPOTrainer
dpo_trainer = DPOTrainer(model=sft_model, train_dataset=dpo_dataset, beta=0.1)
dpo_trainer.train()
Evaluation metrics on a 200-example holdout set:
| Metric | Definition | Target |
|---|
| Tool selection accuracy | Correct tool name chosen | >95% |
| Schema adherence | All required fields present + correct types | >98% |
| Argument accuracy | Argument values match query intent | >92% |
Run evaluation before and after fine-tuning to measure improvement. Expected: 15–20 percentage points improvement on all three metrics compared to few-shot prompting with a general model.
Model serving: wrap the fine-tuned model as an OpenAI-compatible API endpoint using vLLM or llama.cpp. This lets you swap fine-tuned models into existing agent frameworks without changing agent code.