Blog · Collection
High-dimensional statistics and machine learning
Sparse regression and its penalties, dimension reduction, autoencoders and generative models, low-rank matrix recovery and canonical correlation, with applications in genomics.
When the number of variables is comparable to or larger than the sample size , classical methods fail; least squares is not even well defined when . This overview combines the statistical answers with modern machine learning, with many examples from genomics.
Sparse regression
If only a few variables matter, penalized regression can find them. The lasso solves
and its penalty sets many coefficients exactly to zero. It also shrinks the large coefficients it keeps, which biases them. The adaptive lasso reweights the penalty using an initial estimate, and nonconvex penalties such as SCAD and MCP level off for large coefficients, so the effects they select are nearly unbiased. Compositional data, such as microbiome proportions that must add up to one, need constrained versions of these models.
Dimension reduction and representation learning
Principal component analysis finds linear structure. Autoencoders learn nonlinear representations by training a network to reconstruct its input through a narrow bottleneck, and clustering in the learned space is a common way to discover cell types or patient subgroups. Generative models, such as variational autoencoders and generative adversarial networks, learn to produce new samples that resemble the data.
Low-rank structure
Many matrices that arise in practice, such as ratings, gene expression panels or images, are approximately low rank. Matrix completion recovers a whole low-rank matrix from a small fraction of its entries, typically by minimizing the nuclear norm (the sum of the singular values), the matrix counterpart of the penalty. Canonical correlation analysis, and its neural-network extension deep CCA, finds structure shared by two views of the same samples, such as gene expression and protein levels.
In practice
Typical applications include population structure in genotype data, where the leading principal components separate ancestry groups, and genome-wide association studies, which test one genetic variant at a time and so must correct for an enormous number of tests.