Blog · Collection

High-dimensional statistics and machine learning

Sparse regression and its penalties, dimension reduction, autoencoders and generative models, low-rank matrix recovery and canonical correlation, with applications in genomics.

When the number of variables pp is comparable to or larger than the sample size nn, classical methods fail; least squares is not even well defined when p>np > n. This overview combines the statistical answers with modern machine learning, with many examples from genomics.

Sparse regression

If only a few variables matter, penalized regression can find them. The lasso solves

β^=arg⁡min⁡β 12n∥y−Xβ∥22+λ∥β∥1,\hat\beta = \arg\min_\beta\ \frac{1}{2n}\lVert y - X\beta \rVert_2^2 + \lambda \lVert \beta \rVert_1,

and its ℓ1\ell_1 penalty sets many coefficients exactly to zero. It also shrinks the large coefficients it keeps, which biases them. The adaptive lasso reweights the penalty using an initial estimate, and nonconvex penalties such as SCAD and MCP level off for large coefficients, so the effects they select are nearly unbiased. Compositional data, such as microbiome proportions that must add up to one, need constrained versions of these models.

Dimension reduction and representation learning

Principal component analysis finds linear structure. Autoencoders learn nonlinear representations by training a network to reconstruct its input through a narrow bottleneck, and clustering in the learned space is a common way to discover cell types or patient subgroups. Generative models, such as variational autoencoders and generative adversarial networks, learn to produce new samples that resemble the data.

Low-rank structure

Many matrices that arise in practice, such as ratings, gene expression panels or images, are approximately low rank. Matrix completion recovers a whole low-rank matrix from a small fraction of its entries, typically by minimizing the nuclear norm (the sum of the singular values), the matrix counterpart of the ℓ1\ell_1 penalty. Canonical correlation analysis, and its neural-network extension deep CCA, finds structure shared by two views of the same samples, such as gene expression and protein levels.

In practice

Typical applications include population structure in genotype data, where the leading principal components separate ancestry groups, and genome-wide association studies, which test one genetic variant at a time and so must correct for an enormous number of tests.

← All collections