Glossary › Architecture

Transformer

Neural network architecture based on self-attention (Vaswani et al., 2017). Processes all tokens in parallel (not sequential like RNNs). Foundation of GPT, BERT, and all modern LLMs. Key innovation: attention is all you need.

Where this is taught

• Jamba — Hybrid SSM-Transformer — Phase 4: Training LLMs
• The Transformer Block & Full Architecture — Phase 4: Transformers & LLM Foundations
• Transfusion: Autoregressive Text + Diffusion Image in One Transformer — Phase 2: Audio-Visual Models
• The Full Transformer — Encoder + Decoder — Phase 0: Attention Mechanisms
• Build a Transformer from Scratch — The Capstone — Phase 2: Pre-training & Fine-tuning
• Multi-Token Prediction (MTP) — Phase 4: Training LLMs

Related terms

Attention Mechanism · Encoder-Decoder

More in Architecture

Attention Mechanism · RAG (Retrieval-Augmented Generation) · Encoder-Decoder · Mixture of Experts (MoE) · KV Cache

Learn this by building

DeVenture Academy teaches AI engineering through projects — every lesson is paired with a hands-on lab you run on your own machine with real tools.

Create a free account — the opening phases of 24 of 30 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary