Glossary › Evaluation

A/B Testing

Comparing two model versions or prompt variants on real user traffic. Statistical significance requires enough samples. Online metric (e.g. thumbs up) vs offline metric (e.g. BLEU).

Where this is taught

• A/B Testing Models & Batch Processing — Phase 7: Multi-Model Pipeline Orchestration
• A/B Testing LLM Features — GrowthBook, Statsig, and the Vibes Problem — Phase 4: Release, Reliability & Compliance
• Prompt Versioning & Management — Phase 1: Advanced Prompt Engineering for Agents
• Bayes' Theorem — Phase 2: Probability & Statistics
• Offline vs Online Evaluation — Phase 5: Evaluation & Quality for AI Products
• Evaluation Framework Design — Phase 7: AI Evaluation & Observability

More in Evaluation

LLM-as-Judge · Golden Set · Regression Testing · BLEU Score

Learn this by building

DeVenture Academy teaches AI engineering through projects, with hands-on labs you run on your own machine using real tools.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary