Phase 4: Transformers & LLM Foundations · 60 min · Hugging Face Transformers · PEFT/LoRA

Pretraining, Fine-tuning & LoRA

Pretraining buys general competence; fine-tuning rents specific behavior. Know which you actually need.

Hiring signal: Chooses adaptation strategy by cost/benefit and knows when NOT to fine-tune

What you will learn

Explain self-supervised pretraining objectives (causal vs masked LM)
Describe transfer learning: pretrain then adapt
Compare full fine-tuning vs parameter-efficient fine-tuning (LoRA)
Decide among prompting, RAG, and fine-tuning for a goal
Outline data and evaluation needs for a fine-tune

The Problem

"Let's fine-tune a model" is the most over-proposed and under-justified plan in applied AI. People reach for fine-tuning to add knowledge (where RAG is better), to fix a one-off prompt (where prompting is better), or without the data and eval to make it work — burning budget for a worse result. Meanwhile they can't explain how the base model learned anything in the first place. Understanding pretraining, transfer learning, and parameter-efficient fine-tuning lets you pick the right adaptation tool and defend it — a core AI-engineering judgment call.

The Concept

Modern LLMs are built in two stages:

1. Pretraining  : learn general language/world structure from massive text,
                  self-supervised (no human labels). Expensive; done once by labs.
2. Adaptation   : specialize the pretrained model to YOUR task. Cheap-ish; your job.

Your company's product catalog changes daily with new items, prices, and specs. You need the LLM to answer questions about current products. Should you fine-tune or use RAG?

Fine-tuning encodes knowledge into weights — it's expensive to retrain daily and the model can't reflect real-time changes. RAG retrieves current catalog data at inference time, so it's always up-to-date with zero retraining. Rule of thumb: use RAG for knowledge that changes, fine-tune for behavior/style that's stable.

Self-supervised pretraining objectives:

Causal LM (GPT)   : predict the NEXT token  -> great for generation
Masked LM (BERT)  : predict MASKED tokens   -> great for understanding/embeddings
Why it scales      : the "labels" are the text itself -> unlimited training signal

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing, Sources & Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.