Phase 5: Modern LLM Engineering · 65 min · Ollama · vLLM · llama.cpp

Local LLMs & Open-Weight Models

The question isn't 'can I run a model locally?' — it's 'should I?' The answer depends on cost at scale, data privacy, latency, and control.

Hiring signal: Can evaluate and deploy local models — critical for privacy-sensitive and cost-sensitive production scenarios

What you will learn

Decide when to use local vs API models based on cost, privacy, latency, and control
Deploy models with Ollama for prototyping and vLLM for production throughput
Compare quantization formats (GGUF, AWQ, GPTQ) and their quality/speed tradeoffs
Benchmark local model throughput and compute break-even cost vs API pricing
Handle GPU memory constraints and hardware requirements

The Problem

Your company handles medical records. Sending patient data to OpenAI violates HIPAA. Or: you're serving 10M requests/day and the API bill is $80k/month. Or: you need sub-50ms latency for a real-time feature. In each case, API-based LLMs are the wrong tool. You need local models.

The Concept

When to Go Local

Reason	Local Advantage	Tradeoff
Privacy	Data never leaves your servers	You manage infra, security patches
Cost at scale	Fixed cost — GPU rental vs per-token	High upfront cost, needs DevOps
Latency	No network round-trip	Limited by your GPU throughput
Control	Full control over inference, fine-tuning	You own bugs and updates
Customization	Fine-tune on your data	Requires ML expertise

You want to run Llama 3.1 70B locally on a machine with 48GB RAM. The FP16 model is ~140GB. What quantization level would let it fit, and what's the quality tradeoff?

INT4 quantization compresses each parameter to 4 bits (0.5 bytes), so 70B params × 0.5 bytes ≈ 35GB. This fits in 48GB with room for the KV cache. Quality degradation is typically 2-3% on benchmarks — acceptable for most use cases. Tools like llama.cpp and Ollama handle this automatically.

The Local LLM Stack

Ollama — Easiest Local Setup

# Install and run a model in 2 commands
ollama pull llama3.1:8b
ollama run llama3.1:8b "Explain RAG in one paragraph"

# Use Ollama from Python — OpenAI-compatible API
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

response = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Write a haiku about tensors"}]
)
print(response.choices[0].message.content)

vLLM — Production Serving

# High-throughput serving with PagedAttention
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --port 8000 \
  --tensor-parallel-size 1

vLLM vs Ollama
Ollama is for development and prototyping — single requests, easy setup. vLLM is for production — batched requests, PagedAttention for memory efficiency, 10-100x higher throughput. Use Ollama to build, vLLM to serve.

Quantization — Making Models Smaller

Quantization reduces model precision (e.g., 16-bit → 4-bit) to fit in less memory:

Format	Bits	Size (8B model)	Quality Loss	Use Case
FP16	16	~16 GB	None	Best quality, needs big GPU
INT8	8	~8 GB	Minimal	Good balance
GGUF Q4	4	~4.5 GB	Small	Ollama, CPU inference
AWQ	4	~4 GB	Small	vLLM production serving

# Load a quantized model with transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    load_in_4bit=True,  # 4-bit quantization
    device_map="auto"
)

The table above gives ballpark sizes, but the mechanics of quantization are simple enough to try yourself: round each weight to a fixed number of int8 levels within its min/max range, then measure the memory saved and precision lost.

import random
import math

random.seed(0)
weights = [random.uniform(-2.0, 2.0) for _ in range(2000)]  # pretend FP32 weights

def quantize_int8(values: list[float]) -> tuple[list[int], float, float]:
    """Scale floats into the int8 range [-127, 127] and round."""
    max_abs = max(abs(v) for v in values)
    scale = max_abs / 127
    quantized = [round(v / scale) for v in values]
    return quantized, scale, max_abs

quantized, scale, max_abs = quantize_int8(weights)
dequantized = [q * scale for q in quantized]

fp32_bytes = len(weights) * 4
int8_bytes = len(quantized) * 1
mean_abs_error = sum(abs(a - b) for a, b in zip(weights, dequantized)) / len(weights)

print(f"FP32 size: {fp32_bytes} bytes, INT8 size: {int8_bytes} bytes ({fp32_bytes / int8_bytes:.1f}x smaller)")
print(f"Mean absolute quantization error: {mean_abs_error:.5f}")

# Try it: shrink the weight range (e.g. random.uniform(-0.1, 0.1)) and see the error drop

Hardware Requirements

FP16 (blue bars) vs Q4 quantized (line) — quantization cuts VRAM by ~3x.

Cost Comparison: API vs Local

# Break-even analysis
api_cost_per_1k_tokens = 0.0005  # GPT-4o-mini
monthly_requests = 10_000_000
avg_tokens_per_request = 500

monthly_api_cost = monthly_requests * avg_tokens_per_request / 1000 * api_cost_per_1k_tokens
# = $2,500/month

# Local: A100 80GB rental = ~$2/hour × 24 × 30 = $1,440/month
# Break-even: ~1.4M requests/month for A100 to be cheaper

Model Selection Guide (2026)

Use case                      → Model              → Size    → Why
──────────────────────────────────────────────────────────────────────
General chat / coding         → Llama 3.1 8B       → 8B      → Best small open model
High-quality reasoning        → Llama 3.1 70B      → 70B     → Rivals GPT-4 on many tasks
Fast / edge / mobile          → Phi-3 Mini         → 3.8B    → Tiny, runs on phone
Multilingual                  → Qwen 2.5 7B        → 7B      → Strong on non-English
Code generation               → DeepSeek Coder 33B → 33B     → Specialized for code
Long context (128k+)          → Mistral Large      → 123B    → 128k context window
Privacy-critical (HIPAA)      → Llama 3.1 8B local → 8B      → Data never leaves server
Cheap bulk processing         → Llama 3.1 8B Q4    → 4.5GB   → 4-bit, runs on 6GB GPU

Selection criteria (in order):
1. Does it fit your VRAM? (check quantized size)
2. Does it pass your eval suite? (run before committing)
3. Is the license compatible? (Llama = commercial OK, some are research-only)
4. Is the context window sufficient for your use case?
5. Is inference speed acceptable? (tokens/sec on your hardware)

Deployment Architecture: Local + API Fallback

import httpx
from dataclasses import dataclass

@dataclass
class LLMConfig:
    local_url: str = "http://localhost:8000/v1"    # vLLM
    api_url: str = "https://api.openai.com/v1"
    api_key: str = ""
    local_model: str = "meta-llama/Llama-3.1-8B-Instruct"
    api_model: str = "gpt-4o-mini"
    max_retries: int = 2
    timeout: float = 10.0

async def generate_with_fallback(
    prompt: str, config: LLMConfig,
) -> str:
    """Try local model first, fall back to API if it fails."""
    # Try local (vLLM) — cheaper, lower latency, private
    for attempt in range(config.max_retries):
        try:
            resp = httpx.post(
                f"{config.local_url}/chat/completions",
                json={
                    "model": config.local_model,
                    "messages": [{"role": "user", "content": prompt}],
                    "max_tokens": 500,
                },
                timeout=config.timeout,
            )
            resp.raise_for_status()
            return resp.json()["choices"][0]["message"]["content"]
        except (httpx.HTTPError, httpx.TimeoutException) as e:
            print(f"Local attempt {attempt+1} failed: {e}")

    # Fallback to API — more reliable, costs money
    print("Falling back to API...")
    resp = httpx.post(
        f"{config.api_url}/chat/completions",
        headers={"Authorization": f"Bearer {config.api_key}"},
        json={
            "model": config.api_model,
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": 500,
        },
        timeout=30.0,
    )
    return resp.json()["choices"][0]["message"]["content"]

# Architecture:
# Request → [Load balancer] → vLLM (local, primary)
#                            ↘ API (OpenAI, fallback)
# Monitor: track local success rate, latency, cost savings

Benchmarking Local vs API Quality

import time, statistics

def benchmark_models(
    eval_cases: list[dict],  # [{"prompt": "...", "expected": "..."}]
    generate_fn,  # callable: (prompt) -> response
) -> dict:
    """Measure latency and quality of a model on your eval set."""
    latencies = []
    correct = 0

    for case in eval_cases:
        start = time.time()
        response = generate_fn(case["prompt"])
        latencies.append(time.time() - start)

        # Simple quality check: does response contain expected keywords?
        if any(kw in response.lower() for kw in case["expected"].lower().split()):
            correct += 1

    return {
        "n_cases": len(eval_cases),
        "accuracy": correct / len(eval_cases),
        "p50_latency_ms": statistics.median(latencies) * 1000,
        "p95_latency_ms": sorted(latencies)[int(len(latencies) * 0.95)] * 1000,
        "mean_latency_ms": statistics.mean(latencies) * 1000,
    }

# Compare:
# local_results = benchmark_models(eval_set, lambda p: call_vllm(p))
# api_results   = benchmark_models(eval_set, lambda p: call_openai(p))
#
# Report: "Local Llama 8B: 87% accuracy, p95=180ms, $0/1k req
#           API GPT-4o-mini: 91% accuracy, p95=420ms, $0.50/1k req
#           Decision: use local for 90% of traffic, API for hard cases"

Write a function that determines whether running a local model is cheaper than using an API, given monthly request count, tokens per request, API cost per 1k tokens, and monthly GPU rental cost.

~~~
def should_go_local(monthly_requests: int, tokens_per_request: int, api_cost_per_1k: float, gpu_monthly_cost: float) -> bool:
    # Return True if local is cheaper, False if API is cheaper
    pass
~~~
assert should_go_local(10_000_000, 500, 0.0005, 1440) == True
assert should_go_local(100_000, 500, 0.0005, 1440) == False
assert should_go_local(1_000_000, 1000, 0.005, 2000) == True
print("should_go_local OK")

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Ship It, Use It, Key Takeaways, Common Pitfalls, Evaluation, Exercises, Key Terms, Interview Framing — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.