Glossary › Infrastructure

vLLM

High-throughput inference engine using PagedAttention for efficient KV cache management. 2-4× faster than HuggingFace transformers. Supports continuous batching, tensor parallelism, and quantization.

Where this is taught

• Serving LLMs: vLLM, Batching & Throughput — Phase 9: LLMOps & Production Deployment
• Local LLMs & Open-Weight Models — Phase 5: Modern LLM Engineering
• EAGLE-3 Speculative Decoding in Production — Phase 1: Serving Engine Internals
• Hardware-Specialized Inference Compilation — FP8 and NVFP4 on Blackwell — Phase 2: Hardware, Quantization & Placement
• Production Quantization — AWQ, GPTQ, GGUF K-quants, FP8, MXFP4/NVFP4 — Phase 2: Hardware, Quantization & Placement
• Production Serving Stack — KV Offloading and Cache-Aware Routing — Phase 1: Serving Engine Internals

Related terms

Inference · KV Cache

More in Infrastructure

Vector Store · ANN (Approximate Nearest Neighbor) · Inference · Quantization · Speculative Decoding

Learn this by building

DeVenture Academy teaches AI engineering through projects, with hands-on labs you run on your own machine using real tools.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary