Blog · Collection
Statistical inference
Likelihood, testing and confidence intervals, large-sample theory, optimality and shrinkage, and what changes in high dimensions.
Statistical inference is reasoning from a sample back to the process that generated it. This overview builds the classical toolkit carefully and then asks where it breaks.
Models, likelihood and three tasks
A statistical model is a family of distributions for the data. Once the data are observed, the likelihood ranks parameter values by how well they explain what was seen. Almost every procedure answers one of three questions: is a hypothesis about believable (testing), what will the next observation look like (prediction), or what is the value of some feature of the population (estimation)?
Two schools of testing
Fisher’s significance test reports a p-value as a measure of evidence against a single hypothesis. The Neyman–Pearson framework instead fixes a decision rule in advance and controls two error probabilities: rejecting a true null (the size, ) and missing a false one (the power is ). The two are often blended in practice, and keeping them apart clarifies what a p-value can and cannot mean.
Confidence intervals come from inverting tests. A 95% interval is a procedure that covers the true parameter in 95% of repeated samples, which is a statement about the procedure, not a probability statement about any one computed interval.
Large-sample theory
Exact sampling distributions are rare, so most methods lean on asymptotics. Estimators that are consistent and asymptotically normal can be turned into tests and intervals, with the delta method handling smooth transformations. The maximum likelihood estimator is the central example: under regularity conditions,
where is the Fisher information of one observation. M-estimators extend the same argument to any estimator defined by minimizing an average loss, and the bootstrap replaces delicate variance formulas with resampling.
Optimality, and the surprise of shrinkage
The Cramér–Rao bound says that no unbiased estimator can have variance below , so the MLE is asymptotically efficient. Yet efficiency among unbiased estimators is not the whole story. To estimate a mean vector from with , the James–Stein estimator
has smaller expected squared error than itself, for every value of . Shrinking toward a common point borrows strength across coordinates, which is also what a hierarchical Bayesian model does.
High dimensions and the Bayes–frequentist middle ground
With many parameters, new problems appear. Testing thousands of hypotheses calls for controlling the false discovery rate instead of each test’s error rate. Choosing a model and then testing within it invalidates naive p-values, which is the problem of selective inference. Regularization deliberately trades bias for variance. And the comparison between posterior-based rules and frequentist guarantees is a theme that runs through all of modern statistics.