Phase 4: Training LLMs · ~75 minutes · Python
DeepSeek-V3 Architecture Walkthrough
Phase 10 · Lesson 14 named the six architectural knobs every open model turns. DeepSeek-V3 (December 2024, 671B parameters total, 37B active) turns all six and adds four more: Multi-Head Latent Attention, auxiliary-loss-free load balancing, Multi-Token Prediction, and DualPipe training. This lesson reads DeepSeek-V3's architecture top to bottom and derives every parameter count from the published config. By the end you can explain why the 671B/37B ratio is the right bet and why MLA + MoE together beat either alone at the frontier.
What you will learn
- Read the DeepSeek-V3 config top to bottom and explain each field in terms of the six GPT-2 knobs plus four DeepSeek-specific additions.
- Derive the total parameter count (671B), active parameter count (37B), and the components that contribute to each.
- Compute the KV cache footprint of MLA at 128k context and compare to what a same-active-param dense model with GQA would pay.
- State the four DeepSeek-specific innovations (MLA, MTP, auxiliary-loss-free routing, DualPipe) and name which part of the architecture/training stack each one targets.
Introduction
- Type: Learn
- Languages: Python (stdlib, parameter calculator)
- Prerequisites: Open Models: Architecture Walkthroughs, Native Sparse Attention (DeepSeek NSA), Multi-Token Prediction (MTP), DualPipe Parallelism
- Time: ~75 minutes
Objective
Learning objectives
- Read the DeepSeek-V3 config top to bottom and explain each field in terms of the six GPT-2 knobs plus four DeepSeek-specific additions.
- Derive the total parameter count (671B), active parameter count (37B), and the components that contribute to each.
- Compute the KV cache footprint of MLA at 128k context and compare to what a same-active-param dense model with GQA would pay.
- State the four DeepSeek-specific innovations (MLA, MTP, auxiliary-loss-free routing, DualPipe) and name which part of the architecture/training stack each one targets.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Problem, The Concept, Build, Check Yourself, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy