Glossary › Architecture

KV Cache

Stores computed Key and Value tensors from previous tokens during autoregressive generation. Eliminates recomputation of past attention — O(n) per new token instead of O(n²). Memory grows linearly with sequence length.

Where this is taught

• KV Cache, Flash Attention & Inference Optimization — Phase 1: Transformer Architecture
• Multi-Region LLM Serving and KV Cache Locality — Phase 2: Hardware, Quantization & Placement
• Inference Optimization — Phase 3: LLM Architecture
• Jamba — Hybrid SSM-Transformer — Phase 4: Training LLMs
• Open Models: Architecture Walkthroughs — Phase 4: Training LLMs
• Production Quantization — AWQ, GPTQ, GGUF K-quants, FP8, MXFP4/NVFP4 — Phase 2: Hardware, Quantization & Placement

Related terms

Attention Mechanism · Inference

More in Architecture

Attention Mechanism · Transformer · RAG (Retrieval-Augmented Generation) · Encoder-Decoder · Mixture of Experts (MoE)

Learn this by building

DeVenture Academy teaches AI engineering through projects, with hands-on labs you run on your own machine using real tools.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary