Categorical Distributions
Overview
In probability theory and statistics, the categorical distribution (also known as the k-way categorical distribution) models outcomes that are mutually exclusive and collectively exhaustive. Unlike continuous distributions, it operates over a finite set of labeled categories rather than numerical values.
While the random variable itself does not possess a natural ordering, the distribution is fully characterized by a probability vector p = (p₁, p₂, ..., p_K) where each pᵢ ≥ 0 and ∑ pᵢ = 1.
Formal Definition
Let X be a random variable that assumes one of K distinct states {1, 2, ..., K}. X follows a categorical distribution with parameter p if:
P(X = i) = p_i for i = 1, 2, ..., K
The parameter vector p lies on the (K−1)-dimensional probability simplex. The distribution is often denoted as:
X ~ Categorical(p)
n = 1). This distinction is crucial when extending to multiple independent trials.
Key Properties
Moment Calculations
Because outcomes are categorical rather than numeric, traditional mean and variance are defined via indicator variables. Let I_i be an indicator that X = i. Then:
E[I_i] = p_i
V[I_i] = p_i(1 − p_i)
Maximum Entropy
Among all distributions on K categories with fixed probabilities, the categorical distribution maximizes entropy when p_i = 1/K for all i, reducing to the uniform distribution.
Support & Domain
| Property | Value |
|---|---|
| Support | {1, 2, ..., K} |
| Domain | Finite categorical set |
| Parameters | p ∈ [0,1]^K, ∑pᵢ = 1 |
| Entropy | −∑ pᵢ log(pᵢ) |
Related Distributions
The categorical distribution forms a foundational building block for several important discrete distributions:
| Distribution | Relationship | Parameters |
|---|---|---|
| Bernoulli | Categorical with K = 2 | p ∈ [0,1] |
| Binomial | Sum of n independent Bernoulli trials | n ∈ ℕ, p ∈ [0,1] |
| Multinomial | Sum of n independent Categorical trials | n ∈ ℕ, p ∈ simplex |
| Dirichlet | Conjugate prior for categorical p | α ∈ ℝ₊^K |
Applications
Categorical distributions are ubiquitous across quantitative disciplines:
- Machine Learning: Output layer of multi-class classification neural networks (via softmax activation).
- Bayesian Statistics: Likelihood function for discrete observed data; paired with Dirichlet priors for conjugate inference.
- Survey Analysis: Modeling responses with multiple unordered categories (e.g., political affiliation, product preference).
- Genetics: Allele frequency modeling at single-nucleotide polymorphism (SNP) loci with multiple variants.
- NLP: Next-word prediction in language models where vocabulary size defines K.
References & Further Reading
- Johnson, N. L., Kotz, S., & Kemp, A. W. (2005). Univariate Discrete Distributions (3rd ed.). Wiley.
- Gelman, A., et al. (2020). Bayesian Data Analysis (3rd ed.). CRC Press. Chapter 15.
- Murphy, K. P. (2023). Probability Machine Learning: A Unifying View. MIT Press.
- Wikipedia Contributors. (2025). "Multinomial distribution". Wikipedia. Retrieved Oct 2025.