Glossary › Deployment

Throughput

Number of tokens or requests processed per second. Increased by batching, continuous batching (vLLM), and tensor parallelism. Trade-off with latency: higher throughput often means higher per-request latency.

Where this is taught

• Serving LLMs: vLLM, Batching & Throughput — Phase 9: LLMOps & Production Deployment
• Inference Optimization — Phase 3: LLM Architecture
• Local LLMs & Open-Weight Models — Phase 5: Modern LLM Engineering
• Queue-Based Agent Dispatch — Phase 8: Production Agent Infrastructure
• Real-Time Vision — Edge Deployment — Phase 2: Object Detection & Segmentation
• Data Pipelines for Pre-Training — Phase 3: LLM Architecture

Related terms

Latency · Inference

More in Deployment

Latency · Streaming

Learn this by building

DeVenture Academy teaches AI engineering through projects — every lesson is paired with a hands-on lab you run on your own machine with real tools.

Create a free account — the opening phases of 24 of 30 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary