Phase 3: Transformers for NLP · ~60 minutes · Python

Structured Outputs & Constrained Decoding

Ask an LLM for JSON. Get JSON most of the time. In production, "most" is the problem. Constrained decoding turns "most" into "always" by editing the logits before sampling.

Hiring signal: Understanding of structured outputs & constrained decoding internals

What you will learn

Implement structured outputs & constrained decoding from scratch
Understand the math and intuition behind the algorithm
Use production libraries for the same task
Ship a reusable artifact

Introduction

Type: Build Languages: Python Prerequisites: Phase 5 · 17 (Chatbots), Phase 5 · 19 (Subword Tokenization) Time: ~60 minutes

The Problem

A classifier prompts an LLM: "Return one of {positive, negative, neutral}." The model returns "The sentiment is positive — this review is overwhelmingly favorable because the customer explicitly states that they ...". Your parser crashes. Your classifier's F1 is 0.0.

Free-form generation is not a contract. It is a suggestion. A production system needs a contract.

Three layers exist in 2026.

Prompting. Ask nicely. "Return only the JSON object." Works ~80% on frontier models, less on smaller ones.
Native structured output APIs. OpenAI response_format, Anthropic tool use, Gemini JSON mode. Reliable on supported schemas. Vendor-locked.
Constrained decoding. Modify the logits at every generation step so the model cannot emit invalid tokens. 100% valid by construction. Works on any local model.

This lesson builds intuition for all three and names when to reach for which.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Concept, Build It, Pitfalls, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.