The Dirichlet distribution is a multivariate generalization of the beta distribution in the continuous domain. It serves as a conjugate prior for categorical and multinomial distributions in Bayesian statistics, making it a foundational tool in probabilistic machine learning, natural language processing, and compositional data analysis.
Formally, it is defined over the probability simplex, meaning its random variables represent a probability distribution themselves: they are non-negative and sum to one. This property makes it ideal for modeling uncertainty over probability vectors.
Mathematical Definition
Let \( \mathbf{X} = (X_1, X_2, \dots, X_K) \) be a random vector taking values in the \((K-1)\)-dimensional simplex, such that \( X_i \geq 0 \) for all \(i\) and \( \sum_{i=1}^K X_i = 1 \). The probability density function (PDF) of a Dirichlet distribution parameterized by \( \boldsymbol{\alpha} = (\alpha_1, \alpha_2, \dots, \alpha_K) \) where \( \alpha_i > 0 \) is given by:
where \( B(\boldsymbol{\alpha}) \) is the multivariate beta function, serving as the normalization constant:
Here, \( \Gamma(\cdot) \) denotes the gamma function. The parameter \( \alpha_i \) controls the "weight" or concentration of the distribution around the \(i\)-th component. Larger values of \(\alpha_i\) push the distribution's mass closer to \(x_i = 1\), while smaller values push it toward zero.
📌 Key Properties
The Dirichlet distribution is the only distribution whose marginals are all beta-distributed, and whose conditional distributions remain Dirichlet. It is fully characterized by its concentration parameters \(\boldsymbol{\alpha}\).
Statistical Properties
The mean, variance, and covariance of the Dirichlet distribution are derived directly from its concentration parameters. Let \( \alpha_0 = \sum_{j=1}^K \alpha_j \).
| Moment | Formula | Interpretation |
|---|---|---|
| Mean | \( \mathbb{E}[X_i] = \frac{\alpha_i}{\alpha_0} \) | Proportional to relative concentration |
| Variance | \( \text{Var}(X_i) = \frac{\alpha_i(\alpha_0 - \alpha_i)}{\alpha_0^2(\alpha_0 + 1)} \) | Decreases as total concentration \(\alpha_0\) increases |
| Covariance | \( \text{Cov}(X_i, X_j) = -\frac{\alpha_i \alpha_j}{\alpha_0^2(\alpha_0 + 1)} \) | Always negative (components compete to sum to 1) |
⚠️ Important Note
Because \( \sum X_i = 1 \), the covariance matrix is singular. This reflects the inherent dependency between components: an increase in one probability necessarily reduces the others.
Applications
Bayesian Inference
The Dirichlet distribution serves as the conjugate prior for the multinomial distribution. If observations follow a multinomial distribution with unknown probabilities \(\mathbf{p}\), placing a \(\text{Dir}(\boldsymbol{\alpha})\) prior on \(\mathbf{p}\) yields a posterior that is also Dirichlet: \( \text{Dir}(\boldsymbol{\alpha} + \mathbf{n}) \), where \(\mathbf{n}\) is the count vector. This algebraic convenience is central to Bayesian modeling of categorical data.
Natural Language Processing & Topic Modeling
In Latent Dirichlet Allocation (LDA), the Dirichlet distribution models both the topic distribution in documents and the word distribution in topics. Its ability to represent sparse probability vectors makes it highly effective for capturing the linguistic property that documents typically focus on a few topics.
Machine Learning & Deep Ensembles
Modern architectures like Dirichlet Neural Networks explicitly output Dirichlet parameters to quantify epistemic uncertainty. This enables calibrated probabilistic predictions and reliable uncertainty estimation in safety-critical applications.
Compositional Data Analysis
Fields like ecology (species abundance), finance (portfolio weights), and genomics (allele frequencies) deal with data that naturally resides on a simplex. The Dirichlet provides a principled statistical framework for analyzing such closed-composition data.
Related Distributions
- Beta Distribution: The Dirichlet reduces to the beta distribution when \(K=2\).
- Multinomial Distribution: The discrete counterpart; Dirichlet is its conjugate prior.
- Gamma Distribution: If \(Y_i \sim \text{Gamma}(\alpha_i, 1)\) independently, then \(X_i = Y_i / \sum Y_j \sim \text{Dir}(\boldsymbol{\alpha})\).
- Dirichlet Process: A stochastic process whose finite-dimensional distributions are Dirichlet; foundational for nonparametric Bayesian inference.
References & Further Reading
- Blischke, W. R. (1956). "Some Comments on the Dirichlet Density Function." Annals of Mathematical Statistics, 27(1), 152–158.
- Murphy, K. P. (2012). Machine Learning: A Probabilistic Perspective. MIT Press. Chapter 11.
- Dirichlet, P. G. L. (1813). "Bemerkungen über die Theorie der Ketten." Journal für die reine und angewandte Mathematik.
- Aitchison, J. (1986). The Statistical Analysis of Compositional Data. Chapman & Hall.
- Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). "Latent Dirichlet Allocation." JMLR, 3, 993–1022.