Phase 1: Transformer Architecture · ~75 minutes · Python
GPT — Causal Language Modeling
BERT sees both sides. GPT sees only the past. The triangle mask is the most consequential single line of code in modern AI.
Hiring signal: Understanding of gpt — causal language modeling internals
What you will learn
- Implement gpt — causal language modeling from scratch
- Understand the math and intuition behind the algorithm
- Use production libraries for the same task
- Ship a reusable artifact
Introduction
Type: Build Languages: Python Prerequisites: Phase 7 · 02 (Self-Attention), Phase 7 · 05 (Full Transformer), Phase 7 · 06 (BERT) Time: ~75 minutes
The Problem
A language model answers one question: given the first t-1 tokens, what is the probability distribution over token t? Train on that signal — next-token prediction — and you get a model that can generate arbitrary text one token at a time.
To train it end-to-end on a whole sequence in parallel, you need each position's prediction to depend only on earlier positions. Otherwise the model trivially cheats by looking at the answer.
The causal mask does this. It is a single upper-triangular matrix of -inf values added to attention scores before softmax. After softmax, those positions become 0. Each position can attend only to itself and earlier positions. And because you apply it once to the whole sequence, you get N parallel next-token predictions in one forward pass.
GPT-1 (2018), GPT-2 (2019), GPT-3 (2020), GPT-4 (2023), GPT-5 (2024), Claude, Llama, Qwen, Mistral, DeepSeek, Kimi — they are all decoder-only causal transformers with the same core loop. Just bigger, better data, and better RLHF.
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers The Concept, Build It, Use It, Ship It, Exercises, Key Terms, Further Reading — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy