Blog · Collection
Multivariate statistics
Principal components, the multivariate normal, linear and factor models, canonical correlation, copulas, ICA and nonparametric tests.
Multivariate statistics studies many measurements taken on the same units, where the interesting structure lives in how the variables move together.
Covariance and principal components
For a random vector with covariance matrix , principal component analysis finds the orthogonal directions of greatest variance: the eigenvectors of , ordered by their eigenvalues. Projecting onto the first few components gives the best low-rank approximation of the data in squared error, which is why PCA is the first thing to try on a wide data set.
The multivariate normal and linear models
The multivariate normal is the reference model. It is closed under linear maps, marginalization and conditioning, and its conditional distributions have linear means and constant covariances. The general linear model handles several responses at once. Variance models describe how itself can be structured, for example as diagonal, compound symmetric or depending on covariates.
Latent factors and canonical correlations
A factor model explains correlated variables with hidden factors:
where is diagonal. The factors are identified only up to rotation, which is both a nuisance and an opportunity to find an interpretable rotation. Canonical correlation analysis works with two sets of variables measured on the same units and finds pairs of linear combinations, one from each set, that are as correlated as possible.
Beyond normality
Copulas separate the dependence between variables from their marginal distributions. A Gaussian copula, for example, converts each variable to normal scores and models only the correlations among those scores, and a rank-based likelihood makes this work even for ordinal data. Independent components analysis goes further than PCA by looking for directions that are statistically independent rather than merely uncorrelated, which is only possible when the data are not Gaussian. Finally, permutation tests and energy tests compare multivariate distributions without assuming any parametric form.