Phase 1: Serving Engine Internals · ~45 minutes · Python (stdlib · engine-decision tree walker)

Self-Hosted Serving Selection — Matching Engine to Hardware and Scale

Engine selection is a function of hardware, scale, and ecosystem — not a leaderboard read.

Hiring signal: Can operate self-hosted serving selection in production

What you will learn

Pick an engine given hardware (CPU / AMD / NVIDIA Hopper / Blackwell), scale (1 user / 100 / 10,000), and workload (general chat / agent / long-context).
Name the 2026 TGI maintenance-mode status (December 11, 2025) and why it biases new projects toward vLLM or SGLang.
Describe the dev/staging/prod pipeline, including where a GGUF-to-safetensors format conversion sits between stages.
Explain why "CPU-first" points to llama.cpp and "AMD" excludes TRT-LLM.

Introduction

Four engines dominate self-hosted inference in 2026: llama.cpp, Ollama, vLLM, SGLang, with TGI trailing in maintenance mode. llama.cpp is fastest on CPU — widest model support, full control over quantization and threading. Ollama is the dev-laptop one-command install, ~15-30% slower than llama.cpp (Go + CGo + HTTP serialization), 3x throughput gap under prod-like load. TGI entered maintenance mode December 11, 2025 — only bug fixes, ~10% slower raw throughput than vLLM but historically top observability and HF-ecosystem integration. That maintenance status makes it a risky long-term bet — SGLang or vLLM are safer defaults for new projects. vLLM is the general-purpose production default — v0.15.1 (February 2026) adds PyTorch 2.10, RTX Blackwell SM120, H200 optimization. SGLang is the agentic multi-turn / prefix-heavy specialist — 400,000+ GPUs in production (xAI, LinkedIn, Cursor, Oracle, GCP, Azure, AWS). Hardware constraints: CPU-first → llama.cpp. AMD / non-NVIDIA → vLLM is the strongest-supported path (TRT-LLM is NVIDIA-locked). 2026 pipeline pattern: dev = Ollama, staging = llama.cpp, prod = vLLM or SGLang. The engines take different weight formats — GGUF for the llama.cpp family, HF safetensors for the GPU engines — so a format conversion may sit between stages.

Type: Learn Languages: Python (stdlib, engine-decision tree walker) Prerequisites: All Phase 17 lessons covering engines (04, 06, 07, 09, 18) Time: ~45 minutes

Objective

Learning objectives

Pick an engine given hardware (CPU / AMD / NVIDIA Hopper / Blackwell), scale (1 user / 100 / 10,000), and workload (general chat / agent / long-context).
Name the 2026 TGI maintenance-mode status (December 11, 2025) and why it biases new projects toward vLLM or SGLang.
Describe the dev/staging/prod pipeline, including where a GGUF-to-safetensors format conversion sits between stages.
Explain why "CPU-first" points to llama.cpp and "AMD" excludes TRT-LLM.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.