Phase 4: Transformers & LLM Foundations · 60 min · tiktoken · Hugging Face tokenizers · PyTorch

Tokenization, Embeddings & Positional Encoding

A transformer never sees text. It sees token IDs turned into vectors plus a sense of position.

Hiring signal: Understands the input pipeline that explains context limits, costs, and tokenizer bugs

What you will learn

Explain why models use subword tokenization (BPE) not words or characters
Map tokens to embedding vectors and describe what embeddings encode
Explain why transformers need positional information and how it's added
Contrast sinusoidal, learned, and rotary (RoPE) positional encodings
Connect tokenization to context limits, cost, and common bugs

The Problem

Everyone obsesses over attention, but the model's input pipeline — how raw text becomes numbers — is what actually explains the behavior you hit in practice: why you get charged per token, why context windows have hard limits, why a model can't spell or do arithmetic on digits, and why "strawberry" has a weird number of r's according to GPT. These aren't mysteries; they're tokenization. And because attention is order-blind, the pipeline must also inject position. Understanding this layer turns confusing LLM quirks into predictable consequences.

The Concept

The Tokenization Trilemma

A transformer never sees text — it sees numbers. The first step in any NLP pipeline is converting text into a sequence of integer IDs, and the choice of how to split text into tokens determines the model's capabilities, costs, and quirks. There are three options, each with a fatal flaw:

Word-level tokenization splits on spaces: "The cat sat" → ["The", "cat", "sat"]. The problem: English alone has hundreds of thousands of words, and the vocabulary explodes. Worse, the model can't handle words it has never seen — a new name, a misspelling, or a word from another language becomes <UNK>, and the model is blind to it.

Character-level tokenization splits into individual characters: "The cat sat" → ["T", "h", "e", "c", "a", "t", "s", "a", "t"]. The vocabulary is tiny (~100 tokens), but sequences become extremely long. A 1000-word document becomes 5000+ tokens, making attention (which is O(n²)) prohibitively expensive. The model also struggles to learn meaningful patterns from individual characters — a single character carries very little semantic signal.

Subword tokenization (BPE, WordPiece, SentencePiece) is the sweet spot: common words stay as single tokens, rare words are split into reusable pieces. "unbelievable" → ["un", "believ", "able"]. The vocabulary is manageable (~30K-100K tokens), unseen words can always be represented (by decomposing into known subwords), and sequences stay reasonably short. This is why every modern LLM uses subword tokenization.

text  --tokenizer-->  token IDs  --embedding-->  vectors  --(+ positional)-->  to attention
"cats" ->  [4937]      ->  lookup row 4937  ->  [0.1, -0.4, ...]  + position info

Three stages:

1. Tokenization      : split text into subword units, map to integer IDs
2. Token embedding   : each ID indexes a learned vector (its meaning)
3. Positional encoding: add order information, since attention is permutation-invariant

Why Positional Encoding Is Necessary

Self-attention is permutation-invariant — it computes the same attention weights regardless of the order of tokens. To the attention mechanism, "dog bites man" and "man bites dog" are identical bags of tokens. This is clearly wrong: word order carries meaning. Positional encoding solves this by adding a position-dependent signal to each token embedding before attention, so the model can distinguish "dog in position 1" from "dog in position 3."

If you feed token embeddings into a Transformer WITHOUT positional encoding, what goes wrong?

Self-attention is permutation-invariant — it computes the same attention weights regardless of token order. Without positional encoding, the model sees "dog bites man" and "man bites dog" as identical inputs. Positional encoding (sinusoidal, learned, or RoPE) injects order information so the model can distinguish word arrangements.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers Build It, Use It, Ship It, Evaluation, Exercises, Key Terms, Common Pitfalls, Interview Framing, Sources & Further Reading — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.