Home › Courses › Transformers & LLMs from Scratch

Transformers & LLMs from Scratch

Pre-train a real 124M parameter GPT: tokenizer, data pipeline, training and evaluation

5 phases. 40 lessons. 40 labs. 1 capstone. The transformer architecture from the ground up — self-attention from scratch, multi-head attention, positional encoding, the full encoder-decoder, BERT and GPT families, and then building a real LLM: tokenizer from scratch, data pipeline, pre-training a 124M parameter GPT, instruction tuning, RLHF, DPO, and modern architecture innovations (MoE, KV cache, flash attention, speculative decoding, quantization). You pre-train a real LLM, not just fine-tune one.

5 phases · 40 lessons · 40 labs · 1 capstone

Take Deep Learning from Scratch first — this course builds on it.

Outcomes you will have by the end

• A pre-trained 124M parameter GPT — A real LLM you pre-trained from scratch — your tokenizer, your data pipeline, your training loop, your evaluation. Not a fine-tune of someone else's model — your model, built from random initialization.
• A working mental model of how LLMs actually work — Self-attention, positional encoding, pre-training, instruction tuning, RLHF — you built each component from scratch. When you use GPT-4 or Claude, you know exactly what's happening underneath.
• A complete LLM pipeline you can extend — The full lifecycle — tokenize, pre-train, evaluate, instruction-tune, quantize, serve — built as modular components you can modify, extend, or replace for any LLM project.
• Verified certificate of completion — Issued by DeVenture Academy, tied to your completion record. Lists the transformer, pre-training, alignment, and optimization skills you built.

What you will be able to do

Self-Attention & Multi-Head Attention · Positional Encoding (Sinusoidal, RoPE, ALiBi) · BERT & GPT Architecture · BPE Tokenization from Scratch · LLM Pre-Training (124M GPT) · Instruction Tuning (SFT) · RLHF & DPO Alignment · KV Cache, Flash Attention & Speculative Decoding · Quantization (INT8, GPTQ, AWQ, GGUF) · Mixture of Experts (MoE)

Every phase, every lesson, every project

• Attention Mechanisms (6 lessons) — free — Self-attention, multi-head attention, and positional encoding from scratch.
  • Why Transformers — The Problems with RNNs
  • Self-Attention from Scratch
  • Multi-Head Attention
  • Positional Encoding — Sinusoidal, RoPE, ALiBi
  • The Full Transformer — Encoder + Decoder
  • BERT — Masked Language Modeling
• Transformer Architecture (6 lessons) — free — Encoder-decoder, tokenization, and the full transformer from scratch.
  • GPT — Causal Language Modeling
  • T5, BART — Encoder-Decoder Models
  • Vision Transformers (ViT)
  • Audio Transformers — Whisper Architecture
  • Mixture of Experts (MoE)
  • KV Cache, Flash Attention & Inference Optimization
• Pre-training & Fine-tuning (4 lessons) — Pre-training objectives, fine-tuning, and RLHF from scratch.
  • Scaling Laws
  • Build a Transformer from Scratch — The Capstone
  • Attention Variants — Sliding Window, Sparse, Differential
  • Speculative Decoding — Draft, Verify, Repeat
• LLM Architecture (12 lessons) — GPT from scratch, KV cache, quantization, and inference optimization.
  • Tokenizers: BPE, WordPiece, SentencePiece
  • Building a Tokenizer from Scratch
  • Data Pipelines for Pre-Training
  • Pre-Training a Mini GPT (124M Parameters)
  • Scaling: Distributed Training, FSDP, DeepSpeed
  • Instruction Tuning (SFT)
  • RLHF: Reward Model + PPO
  • DPO: Direct Preference Optimization
  • Constitutional AI and Self-Improvement
  • Evaluation: Benchmarks, Evals, LM Harness
  • Quantization: Making Models Fit
  • Inference Optimization
• Training LLMs (12 lessons) — Data pipelines, distributed training, and evaluation.
  • Building a Complete LLM Pipeline
  • Open Models: Architecture Walkthroughs
  • Speculative Decoding and EAGLE-3
  • Differential Attention (V2)
  • Native Sparse Attention (DeepSeek NSA)
  • Multi-Token Prediction (MTP)
  • DualPipe Parallelism
  • DeepSeek-V3 Architecture Walkthrough
  • Jamba — Hybrid SSM-Transformer
  • Async and Hogwild! Inference
  • Speculative Decoding and EAGLE
  • Gradient Checkpointing and Activation Recomputation

The technologies you will use

PyTorch · Hugging Face Transformers · NumPy · tiktoken · Weights & Biases

Roles this course prepares you for

• LLM Engineer ($130k–$200k) — Build, fine-tune, and deploy LLM systems. Understanding pre-training from scratch means you can diagnose training issues, design custom fine-tuning strategies, and optimize inference in ways that API-only engineers can't.
• AI Research Engineer ($150k–$250k) — Implement novel LLM architectures from research papers — new attention mechanisms, training strategies, or alignment techniques. The from-scratch understanding is essential for research-level work.
• NLP Engineer ($120k–$180k) — Build NLP systems — text classification, generation, translation, summarization. Understanding transformers from scratch means you can choose the right model, fine-tune effectively, and debug generation quality issues.

What transformers & LLMs from scratch actually is

It's building the transformer architecture from the ground up — self-attention, multi-head attention, positional encoding, encoder-decoder — and then using that understanding to pre-train a real 124M parameter GPT model. You build the tokenizer, the data pipeline, the training loop, the instruction tuning, and the alignment. By the end, you've built a complete LLM, not just used one.

What you do every day

You build LLM systems — fine-tuning, inference, evaluation. But you understand every component at the implementation level. When generation quality drops, you know whether it's the tokenizer, the model architecture, the training data, or the inference setup. When a new architecture is released, you can read the paper and understand what changed.

Why companies hire for this

Every company wants to use LLMs, but most engineers treat them as APIs. The engineer who pre-trained a GPT from scratch is the one who can build custom fine-tuning pipelines, optimize inference, and implement novel architectures from papers. That's the engineer companies pay top dollar for.

What this course is not

It is not a prompt engineering course. It is not a LangChain tutorial. It is not about using the OpenAI API. This course is about building the model itself — the architecture, the training, the alignment — from scratch. If you want to use LLMs as a black box, take ML & AI Engineering instead.

Common questions

Do I need Deep Learning from Scratch first?

Yes — this course uses PyTorch, autograd, backpropagation, and training loops throughout. Deep Learning from Scratch builds those foundations; this course applies them to the transformer architecture and LLM pre-training.

Can I actually pre-train a 124M parameter model?

Yes — a 124M parameter model is small enough to train on a single GPU (or even a good CPU with patience). The course is designed so the capstone is achievable on consumer hardware, while teaching you the same pipeline used for billion-parameter models.

How is this different from the Transformers phase in ML & AI Engineering?

ML & AI Engineering covers transformers in 6 lessons — enough to fine-tune a pre-trained model. This course spends 20 lessons building attention from scratch, implementing BPE tokenization, pre-training a GPT model, implementing RLHF/DPO, and covering modern architecture innovations. It's the difference between using an LLM and building one.

What do I end up with?

A pre-trained 124M parameter GPT model with your own tokenizer, data pipeline, training loop, instruction tuning, and evaluation — plus a complete LLM pipeline you can extend for any future LLM project. And the ability to read any LLM paper and understand the architecture.

Key terms in this course

RLHF · KV Cache · Mixture of Experts (MoE) · Quantization · Transformer · DPO (Direct Preference Optimization) · Encoder-Decoder · Fine-Tuning

Continue your learning path

Deep Learning from Scratch · Multimodal AI Systems · NLP & Speech Processing · ML & AI Engineering

Start the Transformers & LLMs from Scratch course

Create a free account — the opening phases of 24 of 30 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary