Glossary › Math

Cross-Entropy Loss

Standard loss function for classification: L = -Σ y_i log(p_i). Measures difference between predicted probability distribution and true distribution. Used in language modeling (next-token prediction).

Where this is taught

• Information Theory — Phase 3: Advanced Math for AI
• Probability and Distributions — Phase 2: Probability & Statistics
• Logistic Regression — Phase 1: Classification Algorithms
• Pre-Training a Mini GPT (124M Parameters) — Phase 3: LLM Architecture
• Classification & Logistic Regression — Phase 2: Classical Machine Learning

Related terms

Softmax

More in Math

Softmax · Cosine Similarity · Perplexity

Learn this by building

DeVenture Academy teaches AI engineering through projects, with hands-on labs you run on your own machine using real tools.

Create a free account — the opening phases of 29 of 35 courses are free, no credit card. Or see Pro pricing.

All courses · Pricing · About · FAQ · Glossary