Phase 4: Training LLMs · ~75 minutes · Python

DeepSeek-V3 Architecture Walkthrough

Phase 10 · Lesson 14 named the six architectural knobs every open model turns. DeepSeek-V3 (December 2024, 671B parameters total, 37B active) turns all six and adds four more: Multi-Head Latent Attention, auxiliary-loss-free load balancing, Multi-Token Prediction, and DualPipe training. This lesson reads DeepSeek-V3's architecture top to bottom and derives every parameter count from the published config. By the end you can explain why the 671B/37B ratio is the right bet and why MLA + MoE together beat either alone at the frontier.

Hiring signal: Understanding of deepseek-v3 architecture walkthrough internals

What you will learn

Read the DeepSeek-V3 config top to bottom and explain each field in terms of the six GPT-2 knobs plus four DeepSeek-specific additions.
Derive the total parameter count (671B), active parameter count (37B), and the components that contribute to each.
Compute the KV cache footprint of MLA at 128k context and compare to what a same-active-param dense model with GQA would pay.
State the four DeepSeek-specific innovations (MLA, MTP, auxiliary-loss-free routing, DualPipe) and name which part of the architecture/training stack each one targets.

Introduction

Type: Learn Languages: Python (stdlib, parameter calculator) Prerequisites: Phase 10 · 14 (open-model walkthroughs), Phase 10 · 17 (NSA), Phase 10 · 18 (MTP), Phase 10 · 19 (DualPipe) Time: ~75 minutes

Learning Objectives

Read the DeepSeek-V3 config top to bottom and explain each field in terms of the six GPT-2 knobs plus four DeepSeek-specific additions.
Derive the total parameter count (671B), active parameter count (37B), and the components that contribute to each.
Compute the KV cache footprint of MLA at 128k context and compare to what a same-active-param dense model with GQA would pay.
State the four DeepSeek-specific innovations (MLA, MTP, auxiliary-loss-free routing, DualPipe) and name which part of the architecture/training stack each one targets.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.