Blog · Collection

Bayesian inference

Priors, posteriors and predictive distributions, conjugate families, and the normal model in one and many dimensions.

The main objects

A Bayesian model has two parts: a sampling model X∣θ∼p(x∣θ)X \mid \theta \sim p(x \mid \theta) and a prior θ∼π(θ)\theta \sim \pi(\theta). Everything else follows from the rules of probability:

The posterior combines what the data say with what was believed beforehand. The predictive distribution is often what an application actually needs, because it averages over the remaining uncertainty about θ\theta.

Conjugacy

A family of priors is conjugate to a likelihood if the posterior stays in the same family. The Beta–Bernoulli model is the classic case: with θ∼Beta(α,β)\theta \sim \text{Beta}(\alpha, \beta) and nn coin flips containing ss heads,

θ∣x∼Beta(α+s, β+n−s).\theta \mid x \sim \text{Beta}(\alpha + s,\ \beta + n - s).

The prior behaves like α+β\alpha + \beta earlier flips, which makes its strength easy to reason about. Every exponential family has a conjugate prior, which is why so many textbook models come in matched pairs: Poisson with gamma, multinomial with Dirichlet, normal with normal.

The normal model

For Xi∣θ∼N(θ,τ−1)X_i \mid \theta \sim N(\theta, \tau^{-1}) with known precision τ\tau and prior θ∼N(μ0,τ0−1)\theta \sim N(\mu_0, \tau_0^{-1}), the posterior is normal, with precision and mean

τn=τ0+nτ,μn=τ0 μ0+nτ xˉτ0+nτ.\tau_n = \tau_0 + n\tau, \qquad \mu_n = \frac{\tau_0\,\mu_0 + n\tau\,\bar x}{\tau_0 + n\tau}.

Precisions add, and the posterior mean is a precision-weighted average of the prior mean and the sample mean. When the variance is also unknown, a normal–gamma prior keeps the model conjugate, and the marginal posterior of the mean becomes a Student-t distribution. The multivariate version replaces precisions with precision matrices and the gamma prior with a Wishart prior, and it is the starting point for many hierarchical models.

Why it matters

These conjugate cases are more than exercises. They are the building blocks of Gibbs samplers, in which each full conditional distribution is often one of them. That connects them directly to the samplers in the MCMC playground.

← All collections