Glossary › Infrastructure

Inference

Running a trained model to produce outputs. Distinct from training. Optimizations: quantization, batching, KV cache, speculative decoding. Cost measured in $/M tokens.

Where this is taught

• Model Inversion & Membership Inference Attacks — Phase 3: Adversarial Machine Learning
• Edge Inference — Apple Neural Engine, Qualcomm Hexagon, WebGPU/WebLLM, Jetson — Phase 2: Hardware, Quantization & Placement
• Hardware-Specialized Inference Compilation — FP8 and NVFP4 on Blackwell — Phase 2: Hardware, Quantization & Placement
• Inference Metrics — TTFT, TPOT, ITL, Goodput, P99 — Phase 3: Measurement & Unit Economics
• Inference Platform Economics — Fireworks, Together, Baseten, Modal, Replicate, Anyscale — Phase 0: Platforms & Inference Economics
• Inference Optimization — Phase 3: LLM Architecture

Related terms

Quantization · KV Cache

More in Infrastructure

Vector Store · ANN (Approximate Nearest Neighbor) · Quantization · vLLM · Speculative Decoding

Learn this by building

DeVenture Academy teaches AI engineering through projects, with hands-on labs you run on your own machine using real tools.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary