Blog · Collection
Bayesian inference
Priors, posteriors and predictive distributions, conjugate families, and the normal model in one and many dimensions.
The main objects
A Bayesian model has two parts: a sampling model and a prior . Everything else follows from the rules of probability:
- the joint distribution ;
- the marginal likelihood, or evidence, ;
- the posterior ;
- the posterior predictive distribution of a new observation, .
The posterior combines what the data say with what was believed beforehand. The predictive distribution is often what an application actually needs, because it averages over the remaining uncertainty about .
Conjugacy
A family of priors is conjugate to a likelihood if the posterior stays in the same family. The Beta–Bernoulli model is the classic case: with and coin flips containing heads,
The prior behaves like earlier flips, which makes its strength easy to reason about. Every exponential family has a conjugate prior, which is why so many textbook models come in matched pairs: Poisson with gamma, multinomial with Dirichlet, normal with normal.
The normal model
For with known precision and prior , the posterior is normal, with precision and mean
Precisions add, and the posterior mean is a precision-weighted average of the prior mean and the sample mean. When the variance is also unknown, a normal–gamma prior keeps the model conjugate, and the marginal posterior of the mean becomes a Student-t distribution. The multivariate version replaces precisions with precision matrices and the gamma prior with a Wishart prior, and it is the starting point for many hierarchical models.
Why it matters
These conjugate cases are more than exercises. They are the building blocks of Gibbs samplers, in which each full conditional distribution is often one of them. That connects them directly to the samplers in the MCMC playground.