Glossary › Architecture

Mixture of Experts (MoE)

Architecture with multiple specialized sub-networks (experts) and a gating network that routes tokens to the best expert(s). Increases parameters without proportional compute. Used in Mixtral, GPT-4.

Where this is taught

• Mixture of Experts (MoE) — Phase 1: Transformer Architecture
• Jamba — Hybrid SSM-Transformer — Phase 4: Training LLMs
• Open Models: Architecture Walkthroughs — Phase 4: Training LLMs
• Multimodal RAG and Cross-Modal Retrieval — Phase 4: Multimodal Deployment
• DeepSeek-V3 Architecture Walkthrough — Phase 4: Training LLMs
• DualPipe Parallelism — Phase 4: Training LLMs

Related terms

Transformer

More in Architecture

Attention Mechanism · Transformer · RAG (Retrieval-Augmented Generation) · Encoder-Decoder · KV Cache

Learn this by building

DeVenture Academy teaches AI engineering through projects — every lesson is paired with a hands-on lab you run on your own machine with real tools.

Create a free account — the opening phases of 24 of 30 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary