The Two Cultures of Statistical Modeling

In 2001, Leo Breiman published a paper in Statistical Science titled "Statistical Modeling: The Two Cultures," and it reads less like a research article and more like a long-overdue argument. Breiman had spent years as a consultant before returning to academia, and when he came back and started reading statistical journals, he noticed something that bothered him: every paper started by assuming a model.

Breiman's central claim is that there are two fundamentally different approaches to learning from data, and that the statistical community had been almost exclusively committed to only one of them. The first approach, which he calls the data modeling culture, starts by assuming that the data was generated by a known stochastic process. You pick a model, usually something like linear regression or logistic regression, estimate its parameters from the data, and then draw conclusions from those parameters. Goodness-of-fit tests and residual analysis are used to check whether the model is acceptable. Breiman estimates that roughly 98 percent of statisticians worked this way at the time he was writing.

The second approach is what he calls algorithmic modeling. Here, the mechanism that generated the data is treated as unknown and complex. The goal is not to recover that mechanism but to find any function that predicts the output accurately. You evaluate the function by its predictive accuracy on held-out test data, not by whether it matches an assumed form. Decision trees, neural nets, and support vector machines all belong to this tradition. Breiman estimates that only about 2 percent of statisticians practiced this way, with the rest of the work being done by computer scientists and engineers in the emerging machine learning community.

The problem Breiman identifies with exclusive reliance on data models is straightforward but serious. When a model is fit to data, the conclusions drawn from it are conclusions about the model's mechanism, not about nature's mechanism. If the model is a poor approximation of how the data was actually generated, the conclusions may simply be wrong. Goodness-of-fit tests are supposed to catch this, but Breiman argues they are too weak to be reliable in practice. He cites work by Bickel, Ritov, and Stoker showing that omnibus goodness-of-fit tests have very little power unless the direction of the alternative is precisely specified. In higher dimensions, residual analysis breaks down similarly. William Cleveland, one of its originators, acknowledged in a seminar that residual analysis could not reliably uncover lack of fit in more than four or five dimensions.

Breiman illustrates the consequences with a study on gender discrimination in faculty salaries. A linear regression found the gender coefficient significant at the 5 percent level, and this was accepted as strong evidence of discrimination. But no one seriously checked whether the data could have been generated by a linear model. The conclusion came from the model, and nobody questioned the model. Breiman saw versions of this problem throughout applied statistics and in medical journals, where the Cox model and logistic regression had become de facto standards without serious scrutiny of whether they fit the data.

The paper also introduces three concepts Breiman considers central to understanding algorithmic modeling. The first is the Rashomon effect, named after Kurosawa's film in which the same events are described differently by different witnesses. In modeling, there are often many functions with nearly the same predictive accuracy but very different structures. In a 30-variable regression, for instance, there can be thousands of five-variable subsets with residual sums of squares within one percent of the minimum. Each one tells a different story about which variables matter. Picking one and treating its structure as truth ignores the instability underlying the selection.

The second concept is the Occam dilemma. Simple, interpretable models tend to be less accurate than complex ones. Breiman shows this concretely by comparing single decision trees to random forests across ten classification datasets. Forests are substantially more accurate in every case, sometimes by a factor of two or more, but their predictions emerge from hundreds of interacting trees and are essentially impossible to interpret directly. This creates a genuine tension between wanting to understand a model and wanting it to work.

The third concept concerns dimensionality. Conventional wisdom held that high-dimensional inputs were dangerous and that the number of features should be reduced aggressively. Breiman argues, and demonstrates with support vector machines and shape recognition forests, that increasing dimensionality can actually help when the right features are added, because a higher-dimensional feature space makes it easier to find separating hyperplanes.

Breiman is careful to say he is not arguing against data models entirely. His position is that the choice of modeling approach should be guided by the problem and the data, not by disciplinary convention. The paper ends with a call for statistics to broaden its toolkit, take predictive accuracy seriously as a criterion, and engage with the kinds of large and complex datasets that were becoming common in science and commerce. Given how machine learning has developed in the two decades since, it is fair to say the field moved in exactly the direction he was pointing.

How Random Forests Actually Work

Leo Breiman's 2001 paper "Random Forests" introduces and formalizes a class of ensemble classifiers that has since become one of the most widely used methods in machine learning. The core idea is simple: grow a large number of decision trees, each one built with some randomness injected into the process, and let them vote on predictions. The aggregate tends to be substantially more accurate than any individual tree.

The formal definition Breiman gives is precise. A random forest is a collection of tree classifiers where each tree depends on the values of a random vector that is sampled independently and with the same distribution for all trees. Each tree gets a vote for the most popular class at a given input, and the forest prediction is whichever class receives the most votes. The randomness can be introduced in different ways, and earlier work by researchers including Ho, Amit, and Geman had explored various approaches. Breiman's contribution is to unify these approaches theoretically and to introduce a specific implementation that performs particularly well in practice.

The theoretical foundation comes from two quantities Breiman calls strength and correlation. Strength measures how accurate individual trees are. Formally, it is the expected margin of the random forest classifier, where margin is defined as the amount by which the average vote for the correct class exceeds the average vote for the best competing class. Correlation measures the statistical dependence between the predictions of different trees in the forest. Breiman derives an upper bound on the generalization error that shows the error is bounded by a quantity that increases with correlation and decreases with the square of the strength. The goal of a well-designed random forest is to maximize strength and minimize correlation simultaneously, which is why the randomness in tree construction is not incidental but load-bearing.

The specific algorithm Breiman calls Forest-RI, for random input selection, achieves this by combining two sources of randomness. Each tree is trained on a bootstrap sample, a random draw with replacement from the original training data, so the trees see different versions of the dataset. At each node of each tree, instead of searching over all available features to find the best split, the algorithm selects a random subset of F features and searches only within that subset. The parameter F controls the tradeoff between strength and correlation: larger F allows the trees to find better splits and increases strength, but it also makes trees more similar to each other, increasing correlation. Breiman finds empirically that the results are not very sensitive to the exact value of F, and that selecting the integer part of log base 2 of M plus 1, where M is the total number of features, works well as a default.

One of the most important theoretical properties of random forests is that they do not overfit as more trees are added. Breiman proves this using the Strong Law of Large Numbers: as the number of trees grows, the aggregate vote converges almost surely to a fixed limit. This means that the generalization error converges to a stable value rather than continuing to decrease on training data while increasing on test data, which is the hallmark of overfitting. Adding trees can only help, never hurt.

The empirical results Breiman reports are strong. Across 19 datasets from the UCI repository plus several synthetic datasets, Forest-RI matches or outperforms Adaboost, which was at the time one of the most accurate classifiers available. On the zip-code handwriting recognition dataset, Forest-RI achieves 6.3 percent test error compared to Adaboost's 6.2 percent, while training substantially faster. On several smaller datasets the margins are larger. In regression, where the forest prediction is the average over individual tree predictions rather than a vote, forests outperform bagging in every dataset tested and outperform adaptive bagging in several.

Robustness to noise is another advantage. When 5 percent of training labels are randomly corrupted, Adaboost's error rate roughly triples on some datasets, because its reweighting mechanism concentrates weight on mislabeled examples that are persistently misclassified, distorting the learned model. Forest-RI shows almost no degradation under the same conditions, because it does not concentrate weight on any subset of examples.

The paper also introduces a method for measuring variable importance internally, without requiring a separate analysis. After each tree is trained, the values of each variable are randomly permuted in the out-of-bag examples, those training examples that were not included in the bootstrap sample for that tree, and those permuted examples are passed through the tree. The increase in misclassification rate due to permuting a variable measures how much that variable contributes to accuracy. Variables that matter produce large increases when permuted; irrelevant variables produce no increase. This importance measure is computed as part of a normal training run and requires no additional data.

The out-of-bag estimation scheme generalizes beyond variable importance. Because roughly one-third of the training data is left out of each individual tree, those left-out examples can be used to estimate generalization error, strength, correlation, and the c over s-squared ratio without setting aside a separate validation set. Breiman shows empirically that these out-of-bag estimates are approximately as accurate as estimates from a held-out test set of the same size, which makes random forests practical in settings where data is limited.

What ML Practitioners Know That Textbooks Don't Cover

Pedro Domingos' 2012 paper "A Few Useful Things to Know About Machine Learning," published in Communications of the ACM, is structured as a guide to the practical knowledge that experienced practitioners develop over time but that rarely appears in formal treatments of the subject. It covers twelve lessons, most of which correct misconceptions or add nuance to ideas that introductory courses present in simplified form.

The first and most fundamental lesson concerns what machine learning is actually optimizing for. Every learner consists of three components: a representation, which defines the space of possible models the learner can express; an evaluation function, which scores models against each other; and an optimization method, which searches for the highest-scoring model. Most textbooks are organized by representation, discussing decision trees, neural networks, support vector machines, and so on as separate topics. But the evaluation and optimization components are equally important and often the source of more practical difficulty.

The second lesson is about generalization. The goal of machine learning is not to perform well on training data but to perform well on new data. This distinction sounds obvious but has significant practical implications. Contamination of test data is common and can be subtle: if parameters are tuned using test data, or if preprocessing decisions are made by examining test data, the reported performance will be optimistic. Domingos also notes that in some settings, a locally optimal solution found by greedy search generalizes better than the global optimum, because the global optimum may be overfit to the training data.

The third lesson is one of the more philosophically significant: data alone is not enough. No learner can generalize from training examples to new examples without making some assumptions about the structure of the problem. Wolpert's no free lunch theorems formalize this by showing that no learning algorithm outperforms random guessing when performance is averaged over all possible functions. The algorithms that work in practice do so because their built-in assumptions match the structure of the real-world problems they are applied to, not because they are assumption-free.

On overfitting, Domingos goes beyond the standard description to decompose generalization error into bias and variance. Bias is the tendency to consistently learn the same wrong thing, a property of models whose hypothesis space cannot represent the true function. Variance is the tendency to learn different things on different training sets, a property of flexible models that are sensitive to the specific data they see. The two are in tension: reducing bias typically requires more complex models with higher variance, and vice versa. Domingos also highlights the problem of multiple testing, where running enough experiments makes spurious results look significant by chance, and notes that cross-validation itself can overfit if used to make too many parameter choices.

The curse of dimensionality receives extended treatment. In high dimensions, geometric intuitions from two or three dimensions break down systematically. A nearest-neighbor classifier that works well in low dimensions fails in high dimensions because all points become approximately equidistant. Even with a trillion training examples and 100 features, the data covers only an astronomically small fraction of the input space. This is not merely a theoretical concern: it directly affects the reliability of similarity-based reasoning that underlies most learning algorithms.

Feature engineering, Domingos argues, is more important than algorithm choice. The single biggest determinant of whether a machine learning project succeeds is the quality and relevance of the features used. Raw data often needs to be transformed, combined, or reframed before a learning algorithm can extract useful structure from it. This work is domain-specific and not automatable in general, which is why most of the time in a machine learning project goes into data preparation and feature construction rather than into running the algorithm.

The point about more data beating cleverer algorithms is empirically well-supported. Different learning algorithms tend to converge to similar performance as the amount of training data increases, because they are all ultimately grouping nearby examples into the same class and the main differences are in how they define proximity. A simpler algorithm with more data frequently outperforms a sophisticated algorithm with less data. This has practical implications for how to allocate effort between improving models and collecting more training examples.

On ensembles, Domingos notes that combining multiple models almost always improves on the best individual model, and the margin can be large. The Netflix Prize competition, in which teams competed to improve a movie recommendation system, was ultimately won by a stacked ensemble of over 100 learners, and combining the two finalist ensembles improved performance further still. Model ensembles are distinct from Bayesian model averaging, which weights hypotheses by posterior probability and tends in practice to be dominated by a single high-probability hypothesis, making it equivalent to just selecting that hypothesis.

The final lesson concerns causation. Machine learning algorithms learn statistical associations, not causal relationships. A model that accurately predicts which patients will be readmitted to a hospital does not tell you that intervening on any particular feature will change that outcome. Using predictive models to guide interventions requires additional reasoning, typically involving experimental data or domain knowledge, that the model itself cannot provide.

The Curse of Dimensionality, Made Concrete

One of the recurring themes in Pedro Domingos' "A Few Useful Things to Know About Machine Learning" is that problems that are tractable in low dimensions can become fundamentally different in high dimensions, and that the intuitions developed from low-dimensional experience are often wrong when applied to data with many features. This isn't a minor technical caveat but a core property of high-dimensional geometry with direct consequences for how machine learning algorithms behave.

The basic problem is that volume grows exponentially with dimension. In one dimension, covering 10 percent of the range of a unit interval requires an interval of length 0.1. In two dimensions, covering 10 percent of the area of a unit square requires a square with side length 0.1 raised to the power of 0.5, which is about 0.32. In ten dimensions, covering 10 percent of the volume of a unit hypercube requires a hypercube with side length 0.1 raised to the power of 0.1, which is about 0.8. By ten dimensions, what looks like a local neighborhood in terms of the fraction of data it contains actually spans 80 percent of the range in every dimension. The concept of locality, which most learning algorithms depend on implicitly or explicitly, ceases to be meaningful.

This has immediate consequences for nearest-neighbor methods. The k-nearest neighbor classifier works by finding the k training examples closest to a new test point and predicting the majority class among them. In low dimensions with dense training data, the nearest neighbors are genuinely nearby and represent similar inputs. In high dimensions, the nearest neighbors of any point tend to be nearly as far away as any other point in the dataset. Domingos gives a specific example: if the class depends only on two features but there are 98 irrelevant additional features, the noise from those irrelevant features completely overwhelms the signal in the two relevant ones, and the nearest-neighbor classifier makes essentially random predictions.

The geometry worsens further. In high dimensions, most of the volume of a hypersphere is concentrated near its surface rather than its interior. If you sample points uniformly from a high-dimensional ball, almost all of them will be found in a thin shell near the boundary. If you inscribe a hypersphere inside a hypercube, the fraction of the hypercube's volume that lies inside the hypersphere approaches zero as the number of dimensions increases. These are precise geometric facts with no approximation. They mean that algorithms relying on volumetric arguments, density estimates, or geometric intuitions built in low dimensions cannot be trusted to behave similarly in high-dimensional settings.

The theoretical guarantees that machine learning relies on are also affected. PAC learning bounds, which specify how many training examples are needed to ensure good generalization with high probability, grow with the size of the hypothesis space. For many natural function classes, the hypothesis space is exponential in the number of features. The bounds then imply an exponential sample size requirement, which cannot be satisfied by any realistic dataset. Domingos notes that these bounds are typically very loose in practice and that real algorithms generalize with far less data than the bounds require. But the looseness is itself somewhat uncomfortable: if the theory says you need millions of examples and you're working with thousands, you don't have a proof that your model will generalize, you just have empirical hope that it does.

There is a partial counterweight to the curse of dimensionality that Domingos calls the blessing of nonuniformity. Real-world data is almost never distributed uniformly across the input space. Images of handwritten digits form a low-dimensional manifold within the space of all possible pixel arrays: not all combinations of pixel values correspond to plausible digit images. Speech signals occupy a small subspace of all possible audio waveforms. Natural language follows syntactic and semantic constraints that rule out most possible sequences of words. Learning algorithms can exploit this lower effective dimensionality without explicitly modeling the manifold structure, which is part of why methods like nearest neighbors work better on real data than the curse-of-dimensionality argument would predict.

Dimensionality reduction methods make this structure explicit. Principal component analysis projects the data onto the directions of maximum variance, reducing the number of dimensions while retaining most of the variation in the data. Nonlinear methods like Isomap attempt to find low-dimensional representations that preserve the intrinsic geometry of the data even when the manifold is curved. These methods can substantially improve the performance of downstream algorithms by removing irrelevant dimensions and reducing the sparsity problem.

The practical implication for anyone working with high-dimensional data is to approach it with more skepticism about apparent patterns. In high dimensions, spurious correlations are common because there are many possible pairs of variables and some will appear related by chance. Models can appear to fit training data well while generalizing poorly, not because of a subtle algorithmic failure but because the high-dimensional space is inherently sparse and the training data covers too little of it to make confident inferences.

Principal Component Analysis From the Ground Up

Jonathon Shlens' "A Tutorial on Principal Component Analysis" takes a technique that is often presented as a computational black box and builds it up carefully from geometric intuition through linear algebra to a working implementation. The paper is worth reading carefully because PCA is one of those methods that gets applied widely but is genuinely understood by fewer people than use it.

The motivating problem Shlens introduces is a physics experiment. A ball on a spring oscillates back and forth along a single axis. Three cameras observe the ball from different angles, none of which aligns with the direction of motion. Each camera records a two-dimensional projection of the ball's position, giving six total measurements for every moment in time. The underlying system has one degree of freedom, but the experimenter sees six correlated, noisy measurements and has no way of knowing in advance that this is the case. The goal of PCA is to identify, from the data alone, that most of the variation can be explained by a single direction, and to find what that direction is.

The formal setup treats the dataset as an m by n matrix X, where m is the number of measurement types and n is the number of observations. Each column of X is a single snapshot of the system. The covariance matrix of the data is defined as the m by m matrix CX, equal to one over n times X multiplied by its transpose. The diagonal entries of CX are the variances of the individual measurements. The off-diagonal entries are the covariances between pairs of measurements, capturing the redundancy in the data. Large off-diagonal values indicate that two measurements are highly correlated and therefore partially redundant.

PCA seeks a linear transformation Y equals PX such that the covariance matrix of Y is diagonal. A diagonal covariance matrix means the transformed variables are uncorrelated, which is the formal statement of removing redundancy. The rows of P define a new set of basis vectors for the data, and the transformed data Y expresses each observation in terms of those new basis vectors. The principal components are the rows of the optimal P.

The mathematical result is that the optimal P is the matrix whose rows are the eigenvectors of CX. This follows from a property of symmetric matrices: any symmetric matrix can be diagonalized by an orthogonal matrix of its eigenvectors. Since CX is symmetric by construction, its eigenvectors form such a matrix. When P is set to the transpose of this eigenvector matrix, the expression for the covariance of Y simplifies to the diagonal matrix of eigenvalues. The eigenvalues give the variance of the data along each principal component, which provides a natural way to rank the components by importance.

Shlens also derives the connection between PCA and singular value decomposition. Define a new matrix Y as one over the square root of n times the transpose of X. Then the matrix Y transpose times Y equals the covariance matrix CX. Computing the SVD of Y gives a factorization U times sigma times V transpose. The columns of V are the eigenvectors of Y transpose Y, which are exactly the eigenvectors of CX, which are the principal components. The singular values on the diagonal of sigma are the square roots of the eigenvalues of CX and are proportional to the standard deviations of the data along each principal component. Most numerical implementations of PCA use SVD rather than direct eigendecomposition because SVD is more numerically stable when the covariance matrix is ill-conditioned.

The practical procedure is straightforward. Organize the data as an m by n matrix with measurements as rows. Subtract the mean of each row so that each measurement type has zero mean. Compute either the eigenvectors of the covariance matrix or the SVD of the mean-subtracted data matrix. The resulting eigenvectors, ordered by their eigenvalues from largest to smallest, are the principal components. Project the original data onto the top k principal components to get a reduced representation that captures the k most important directions of variation.

The assumptions embedded in this procedure are worth stating explicitly. PCA assumes that the important structure in the data lies along directions of large variance, which requires that the signal-to-noise ratio is reasonably high. It assumes that the relevant relationships are linear, meaning the important directions are linear combinations of the original variables. It assumes that the principal components should be orthogonal to each other, which simplifies the algebra and makes the solution unique but imposes a geometric constraint on the answer.

Shlens discusses two categories of failure. The first involves nonlinear structure. If a person is moving in a circle, the trajectory can be described by a single variable, the angle, but that angle is a nonlinear function of the Cartesian coordinates. PCA will find two principal components corresponding to the x and y directions rather than recovering the angle, because it can only represent linear combinations. Kernel PCA addresses this by applying a nonlinear transformation to the data before computing principal components. The second category involves non-Gaussian data. PCA minimizes second-order dependencies, meaning correlations, but does not remove higher-order dependencies. Independent component analysis, which maximizes statistical independence rather than decorrelation, handles cases where the important structure is captured by higher-order statistics. Shlens notes that when a data set is Gaussian distributed, removing second-order dependencies is sufficient, because Gaussian distributions are fully characterized by their first two moments.