5.2 Choosing the Prior

In Bayesian inference, the prior distribution p(θ) encodes beliefs about model parameters before observing data. Unlike frequentist approaches, which treat parameters as fixed unknowns, Bayesian methods treat them as random variables. This flexibility is powerful, but it introduces a critical design choice: how should we specify the prior? [1]

An ill-chosen prior can bias estimates, obscure data signals, or cause computational instability. Conversely, a well-specified prior regularizes models, incorporates expert knowledge, and improves interpretability. This section outlines a structured framework for prior selection across scientific and engineering contexts.

Selection Framework

Choosing a prior should never be an afterthought. A robust workflow considers three dimensions:

1. Domain Knowledge: What do we know about the parameter's plausible range, scale, or distribution? Even weak constraints (e.g., positivity, bounded support) matter.
2. Model Purpose: Is the goal prediction, causal inference, or parameter estimation? Predictive models tolerate weaker priors; causal claims require careful sensitivity analysis.
3. Computational Constraints: Conjugate priors simplify exact inference, while hierarchical or non-conjugate priors may require MCMC or variational methods.

Prior Categories

Priors are typically classified by the strength and origin of the information they encode:

• Informative Priors: Encode substantial domain knowledge (e.g., a normal prior centered on a literature-reported effect size). Useful when data is sparse.
• Weakly Informative Priors: Provide mild regularization without dominating the likelihood. Example: Normal(0, 10) for standardized regression coefficients.
• Reference/Non-informative Priors: Designed to minimize prior influence (e.g., Jeffreys prior, uniform over support). Caution: these can be improper or yield counterintuitive results in hierarchical models.
• Empirical Priors: Estimated from auxiliary data or cross-validation. Common in empirical Bayes and meta-learning settings.

Gelman [3] strongly advocates for weakly informative priors in most applied settings, as they prevent extreme estimates while allowing data to dominate when sample sizes are large.

Practical Guidelines

• Standardize predictors before assigning priors to regression coefficients. This ensures priors are on a comparable scale.
• Use half-Cauchy or exponential for variance components and standard deviations, rather than uniform, to avoid boundary artifacts.
• Prefer hierarchical priors when modeling grouped data. They automatically induce partial pooling and shrinkage.
• Run prior predictive checks: Simulate data from the prior (and likelihood) to verify that implied outcomes are plausible before seeing real data.
• Document assumptions: Explicitly state why a prior was chosen and how it aligns with domain knowledge.

Common Pitfalls

When the likelihood is flat (low information), the posterior closely mirrors the prior. Common mistakes include:

• Flat priors on unbounded domains: Uniform(−∞, ∞) is improper and can lead to improper posteriors.
• Ignoring parameterization effects: A uniform prior on σ is not uniform on σ². Jacobian adjustments matter.
• Base rate neglect: Assigning equal probability to rare and common events without accounting for known frequencies.
• Overlooking sensitivity: Failing to test how results change under alternative reasonable priors.

Applied Example: Beta-Binomial Model

Suppose we model a success rate θ in a clinical trial. We observe y = 8 successes in n = 20 trials.

• Uninformative: θ ~ Beta(1, 1) (Uniform). Posterior: Beta(9, 13). Mean = 0.41.
• Weakly Informative: θ ~ Beta(2, 2). Encodes mild belief in non-extreme rates. Posterior: Beta(10, 14). Mean = 0.42.
• Informative: Literature suggests success rates near 0.5 with moderate certainty. θ ~ Beta(5, 5). Posterior: Beta(13, 17). Mean = 0.433.

With n=20, all priors yield similar posteriors. With n=2, the informative prior heavily shrinks estimates toward 0.5. This illustrates the trade-off between regularization and data fidelity.

Summary

Choosing a prior is both an art and a science. Start with weakly informative defaults that respect parameter constraints and scale. Incorporate domain knowledge judiciously, validate through prior predictive simulation, and always report sensitivity analyses. In modern Bayesian workflows, the prior is not a nuisance—it is a core component of transparent, reproducible inference.

Next: 5.3 Posterior Computation & MCMC Diagnostics