Glossary › Architecture

Attention Mechanism

Computes weighted relationships between all token pairs in a sequence. Q·K^T produces attention scores, softmax normalizes them, and multiplication with V gives context-aware representations. O(n²) in sequence length.

Where this is taught

• Attention Mechanism — The Breakthrough — Phase 1: Text Representation
• The Attention Mechanism from Scratch — Phase 4: Transformers & LLM Foundations
• Attention: Intuition & Visualization — Phase 4: Transformers & LLM Foundations

Related terms

Transformer

More in Architecture

Transformer · RAG (Retrieval-Augmented Generation) · Encoder-Decoder · Mixture of Experts (MoE) · KV Cache

Learn this by building

DeVenture Academy teaches AI engineering through projects — every lesson is paired with a hands-on lab you run on your own machine with real tools.

Create a free account — the opening phases of 24 of 30 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary