Phase 4: Training LLMs · ~45 minutes · Python

Open Models: Architecture Walkthroughs

You built a GPT-2 Small from scratch in Lesson 04. Frontier open models in 2026 are the same family with five or six concrete changes. RMSNorm instead of LayerNorm. SwiGLU instead of GELU. RoPE instead of learned positions. GQA or MLA instead of full MHA. Mixture-of-Experts at scale. The math you already know covers 95% of them. This lesson reads Llama 3, DeepSeek-V3, Mixtral, Qwen, and Gemma side by side and names the exact line where each architecture diverges.

Hiring signal: Understanding of open models: architecture walkthroughs internals

What you will learn

Read the config.json of Llama 3, Mistral, Mixtral, Gemma 2, Qwen 2.5, and DeepSeek-V3 and explain every field
Name the specific architectural change each model made versus GPT-2 Small and justify it from first principles
Compute parameter count, KV cache size, and activation memory for any open model from its config alone
Pick the right open model for a deployment target given latency, memory, and capability constraints

Introduction

Type: Learn Languages: Python (stdlib) Prerequisites: Phase 10, Lessons 04, 05, 12 (Pre-training, Scaling, Inference) Time: ~45 minutes

Learning Objectives

Read the config.json of Llama 3, Mistral, Mixtral, Gemma 2, Qwen 2.5, and DeepSeek-V3 and explain every field
Name the specific architectural change each model made versus GPT-2 Small and justify it from first principles
Compute parameter count, KV cache size, and activation memory for any open model from its config alone
Pick the right open model for a deployment target given latency, memory, and capability constraints

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build It, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.