Transformers & LLMs from Scratch
Pre-train a real 124M parameter GPT — tokenizer, data pipeline, training, and evaluation
5 phases. 20 lessons. 20 labs. 1 capstone. The transformer architecture from the ground up — self-attention from scratch, multi-head attention, positional encoding, the full encoder-decoder, BERT and GPT families, and then building a real LLM: tokenizer from scratch, data pipeline, pre-training a 124M parameter GPT, instruction tuning, RLHF, DPO, and modern architecture innovations (MoE, KV cache, flash attention, speculative decoding, quantization). You pre-train a real LLM, not just fine-tune
- Lessons: —
- Labs: —
- Projects: —
- Level: Beginner
Curriculum
- Attention Mechanisms — Self-attention, multi-head attention, and positional encoding from scratch.
- Transformer Architecture — Encoder-decoder, tokenization, and the full transformer from scratch.
- Pre-training & Fine-tuning — Pre-training objectives, fine-tuning, and RLHF from scratch.
- LLM Architecture — GPT from scratch, KV cache, quantization, and inference optimization.
- Training LLMs — Data pipelines, distributed training, and evaluation.
Skills You Will Learn
- Self-Attention & Multi-Head Attention
- Positional Encoding (Sinusoidal, RoPE, ALiBi)
- BERT & GPT Architecture
- BPE Tokenization from Scratch
- LLM Pre-Training (124M GPT)
- Instruction Tuning (SFT)
- RLHF & DPO Alignment
- KV Cache, Flash Attention & Speculative Decoding
- Quantization (INT8, GPTQ, AWQ, GGUF)
- Mixture of Experts (MoE)
Related Courses
Browse all courses · View pricing · DeVenture Academy