Linear regression: the honest workhorse

Least squares is a method: give it points, it hands you a line. Linear regression is a model: it makes a claim about how the world generated those points, and then uses least squares to estimate the claim's parameters. The distinction sounds pedantic until you need something a bare line can't give you — a confidence interval, a p-value, a defensible sentence like "each extra year of schooling is worth about eight percent more income, holding experience fixed." Those all come from taking the line seriously as a story about the data, not just a curve that happens to pass through the middle.

Regression is the oldest workhorse in the stable. Francis Galton coined the term in the 1880s while studying heredity — he noticed that the sons of tall fathers were tall, but less tall on average, drifting back toward the population mean. He called it "regression toward mediocrity," and the unflattering name stuck to the whole technique. A century and a half later it's still the first thing anyone reaches for, and often the last thing they should have needed. Let's earn that trust.

The model, and what it claims

Here is the entire assertion, in one line:

y = X\beta + \varepsilon.

Read it slowly, because every symbol is a commitment. y is the n-vector of outcomes you observed. X is the n \times p design matrix — one row per observation, one column per feature, with a column of ones absorbing the intercept. \beta is the p-vector of true coefficients, fixed numbers that exist out in the world whether or not you ever learn them. And \varepsilon is the noise: the part of y that the features genuinely cannot explain.

What does this model claim? That the expected outcome is a linear combination of the features — \mathbb{E}[y \mid X] = X\beta — and that everything else is zero-mean scatter around that plane. What does it not claim? It does not say the world is actually linear; it says a linear function is your best summary of the conditional mean. It does not say \varepsilon is small, only that it's unbiased noise with no leftover pattern. And crucially, "linear" refers to linearity in the parameters, not the features. You can throw x^2, \log x, or \sin x into the columns of X and still be doing linear regression — the model bends happily through curved data as long as it stays a straight-line combination of whatever columns you hand it.

Fitting it is least squares

Given data, how do you find \beta? You don't — you estimate it, and the estimate is exactly the least-squares solution. Choose the \hat\beta that minimizes the sum of squared residuals \lVert y - X\beta \rVert^2, set the gradient to zero, and you land on the normal equations X^\top X \hat\beta = X^\top y, whose solution is

\hat\beta = (X^\top X)^{-1} X^\top y.

That derivation — both the calculus route and the lovely geometric one where \hat\beta turns out to be an orthogonal projection — is the subject of its own essay on least squares, so I won't repeat it here. What matters for us is the shift in interpretation. In least squares, \hat\beta is just "the coefficients of the best line." In regression, \hat\beta is an estimator: a random quantity, because it's computed from noisy data, that we hope lands near the true \beta. Ask how near, and you've walked straight into the theorem that makes regression respectable.

Gauss–Markov: why OLS is BLUE

Here is the promise that made ordinary least squares (OLS) the default for two centuries. Suppose four things hold: the model is correctly specified (\mathbb{E}[\varepsilon] = 0), the errors have constant variance (\mathrm{Var}(\varepsilon_i) = \sigma^2 for all i), the errors are uncorrelated (\mathrm{Cov}(\varepsilon_i, \varepsilon_j) = 0 for i \ne j), and X has full column rank. Then the Gauss–Markov theorem states:

Among all estimators that are linear in y and unbiased, the OLS estimator \hat\beta has the smallest variance.

That's the acronym: BLUE, the Best Linear Unbiased Estimator. Unpack each word, because the theorem gives you a lot and asks for surprisingly little. Linear means the estimator is some fixed matrix times y — and \hat\beta = (X^\top X)^{-1}X^\top y certainly is. Unbiased means it's right on average: \mathbb{E}[\hat\beta] = \beta, no systematic lean in any direction. Best means minimum variance — of every estimator satisfying the first two properties, OLS is the one that jitters least from sample to sample.

The intuition is worth holding onto. Notice the theorem never mentioned Gaussian noise. Maximum likelihood needed a bell curve; Gauss–Markov does not. It asks only for zero mean, constant variance, and no correlation — the first two moments of the noise, nothing about its full shape. That's a remarkably cheap price for a strong conclusion, and it's exactly why OLS shows up everywhere: you rarely know the distribution of your errors, but you can often believe they're roughly patternless with roughly constant spread. When even that's too much to assume, the "best" guarantee weakens — but the estimator still works, which is more than most methods can say.

What a coefficient means

Fit the model and you get a number for each feature. What does \hat\beta_j actually say? Precisely this: the expected change in y for a one-unit increase in feature j, holding all other features fixed. That last clause is the entire point of multiple regression, and the most misread phrase in applied statistics.

"Holding others fixed" is not a thought experiment you run afterward — it's baked into the arithmetic. The coefficient on x_j is what you get by first stripping out of both y and x_j everything the other features can explain, and then regressing the leftovers against each other. So \hat\beta_j measures the effect of the part of x_j that is unique to it, the part its companions can't account for. This is why a coefficient can flip sign when you add a variable: you haven't found a contradiction, you've changed the question from "what's the raw association?" to "what's the association net of this other thing?" Ice-cream sales predict drownings beautifully — until you hold temperature fixed, and the coefficient collapses to nothing. The regression was never wrong; you were just asking it a different question than you thought.

The four assumptions, in plain terms

Gauss–Markov's guarantee rests on assumptions, and it pays to know them by their street names and their failure modes.

Linearity. The conditional mean really is X\beta. If the true relationship curves and you fit a straight line, the model is biased — wrong on average, no matter how much data you gather. The fix is often cheap: add a squared or log term and you're modeling the curve while staying "linear."

Independent errors. One observation's noise says nothing about the next one's. This breaks constantly in time series (today's error looks like yesterday's) and clustered data (students in the same classroom share shocks). When it fails, \hat\beta stays unbiased but your standard errors lie — usually too small, so you'll believe results are significant when they're noise.

Homoscedasticity. The Greek mouthful just means "same scatter" — \mathrm{Var}(\varepsilon_i) is constant across observations. Its opposite, heteroscedasticity, is the fan shape you see when the spread grows with x (rich households vary more in spending than poor ones). Again the coefficients survive, but the error bars are wrong, and the "best" in BLUE quietly evaporates — a weighted fit would beat you.

No perfect collinearity. No feature is an exact linear combination of the others, which is what keeps X^\top X invertible — the same rank condition that the four fundamental subspaces make precise. Include weight in both kilograms and pounds and the math simply can't run — there's no unique way to split credit between two columns carrying the same information. Near-collinearity is subtler and nastier: the inverse exists but is numerically explosive, so coefficients swing wildly on the slightest change in data. The individual \hat\beta_j become untrustworthy even while the overall predictions stay fine.

Notice a pattern: violating linearity or collinearity attacks the estimates themselves, while violating independence or homoscedasticity spares the estimates but corrupts your sense of how sure to be. Knowing which kind of trouble you're in tells you whether to fix the model or just fix the standard errors.

Bias and variance: the two ways to be wrong

Every assumption above circles one central tension, the deepest idea in the whole subject. Your model's expected error decomposes cleanly into two competing pieces:

\underbrace{\text{Bias}^2}_{\text{too rigid}} + \underbrace{\text{Variance}}_{\text{too flexible}} + \underbrace{\sigma^2}_{\text{irreducible}}.

Bias is error from wrong assumptions — a model too simple to capture the truth. Fit a straight line to a parabola and you underfit: the line is stable (it barely moves if you resample the data) but stubbornly, systematically off. Variance is error from oversensitivity — a model so flexible it fits the noise. Snake a high-degree polynomial through every point and you overfit: it nails your sample perfectly and predicts the next one terribly, because it memorized the noise instead of learning the signal.

The two pull in opposite directions, and that's the whole game. Add features or complexity and you cut bias but raise variance; strip them away and you do the reverse. Total error traces a U — steep on the left where you're too simple, steep on the right where you're too clever, with a sweet spot in between that no amount of staring at the training data will reveal to you (you need held-out data for that). Plain OLS sits at a fixed point on this curve; the moment you want to slide along it — trading a little bias for a lot less variance — you're reaching for regularization, cross-validation, and the machinery of the rest of this series.

See the estimator wobble

All of this — "estimator, not answer," variance, the fanning confidence band — is easier to feel than to read. The linear regression playground generates noisy points from a known line, fits them with OLS, and lets you hit resample to draw a fresh sample and watch the fit wobble around the truth: never landing exactly on it, but never leaning systematically to one side either — unbiasedness and variance in the same picture. Crank the noise up and the fit swings harder; add more data and it settles down. Turn on the sampling cloud to see forty fits at once, a fan of lines tightest where the data is dense and loosest at the edges — the confidence band drawn the honest way, one sample at a time.

Where this goes

Linear regression is the trunk from which most of supervised learning branches. Keep the linear predictor X\beta but squash it through a curve so the output becomes a probability, and continuous regression turns into classification — that's logistic regression. Keep the model but ditch the one-shot matrix inverse, which chokes when X has millions of rows, and roll downhill to the same minimum instead — that's gradient descent. The line Galton drew through his fathers and sons is still doing quiet, honest work under nearly everything that came after.