Phase 4: Transformers & LLM Foundations · 70 min · Hugging Face Transformers · PEFT/LoRA · PyTorch

Hands-On LoRA Fine-tuning

Fine-tuning doesn't have to mean full weights. Train a tiny adapter and see the difference.

Real-world application: Has actually run a LoRA fine-tune end-to-end: data prep, training, adapter merge, eval vs baseline

What you will learn

Explain the low-rank decomposition behind LoRA and the role of rank r and alpha
Prepare a small dataset in the right format for a supervised fine-tune
Run a LoRA fine-tune of a small model with the PEFT library
Evaluate the adapter against the base model with a held-out set
Decide when LoRA is enough versus when full fine-tuning is justified

The Problem

Most teams over-spend on fine-tuning because they default to full fine-tuning and cloud GPUs. Parameter-efficient fine-tuning (PEFT), and especially LoRA, lets you specialize a model for a fraction of the compute and storage. The catch: you still need the right data format, a sane eval, and a clear baseline. In this lesson you will actually fine-tune a small model with LoRA and measure it against the base model.

The Concept

Why Low-Rank Adaptation Works

The key insight behind LoRA is empirical: when you fine-tune a large pretrained model, the weight updates have low intrinsic rank. This means that even though the weight matrices are huge (e.g., 4096×4096), the meaningful changes during fine-tuning live in a much smaller subspace — you don't need to update every parameter to adapt the model to a new task.

LoRA exploits this by decomposing the weight update into two small matrices: instead of updating W (a d×d matrix with d² parameters), you freeze W and learn B·A where A is r×d and B is d×r, with r (the rank) much smaller than d. The effective update is W + (α/r)·B·A, and you only train A and B — reducing trainable parameters from d² to 2·r·d. For d=4096 and r=8, that's from 16.7M to 65K parameters — a 256x reduction.

Full fine-tuning        LoRA (Low-Rank Adaptation)
─────────────────       ─────────────────────────
Update all weights        Freeze base weights
Expensive storage         Train small A + B matrices
Risk of catastrophic      Reversible, easy to swap
forgetting              Often 1-10% of parameters

The practical benefits are significant: you can store multiple task-specific adapters (each a few MB) instead of multiple full model copies (each several GB), you can swap adapters at inference time, and training is faster because fewer parameters need gradients. The base model stays frozen, so there's no catastrophic forgetting of the pretrained knowledge.

LoRA rewrites a weight update as a low-rank decomposition. For a pretrained weight matrix $W_0$, the adapted forward pass is:

h = W_0 * x + (alpha / r) * B * A * x

Where $r$ (rank) is small, $A$ is initialized with a random Gaussian, and $B$ is initialized to zero. Only $A$ and $B$ are trained.

You are choosing between LoRA rank 4 and rank 64 for a small classification adapter. Which statement is most accurate?

Rank controls the number of trainable parameters in the adapter. A higher rank increases capacity but also memory and overfitting risk. For simple tasks, a low rank (4-8) is often sufficient and generalizes better.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card.