The shadow of a vector

Stand a pencil upright on your desk in the late afternoon and watch the shadow it throws across the wood. The pencil lives in three dimensions; its shadow is flattened onto two. And here's the thing everybody's eye knows without being told: the shadow is the pencil's honest stand-in on the desk — the closest the desk can come to holding the pencil itself. Slide the sun overhead and the shadow shrinks to a dot directly beneath the tip; that dot is the point on the desk nearest to where the pencil actually is. Every projection in linear algebra is this shadow, made exact. And once it's exact, it answers a question you can't dodge in any applied subject: when the thing you want isn't reachable, what's the closest thing that is?

We left the four fundamental subspaces on exactly this cliff. A system Ax = b has a solution only when b lives in the column space C(A) — the set of reachable outputs. When b misses that subspace, elimination just hands you a contradiction and shrugs. Projection is what you do instead of shrugging: you give up on hitting b exactly and settle for the nearest point of C(A), the foot of the perpendicular. This essay builds that machinery from the ground up, and it's the geometry that least squares and linear regression quietly stand on.

The inner product: a machine for measuring agreement

Before shadows, angles; and before angles, the inner product (or dot product). For two vectors in \mathbb{R}^n,

a^\top b = a_1 b_1 + a_2 b_2 + \cdots + a_n b_n,

a single number that packs in both length and angle. Set b = a and you get a^\top a = \lVert a \rVert^2, the squared length. And the reason the inner product matters for us is the identity

a^\top b = \lVert a \rVert\,\lVert b \rVert \cos\theta,

where \theta is the angle between the two vectors. Read it as a measure of agreement: it's largest when the vectors point the same way, negative when they point apart, and — the case we'll lean on hardest — exactly zero when they're at right angles, because \cos 90^\circ = 0. Two vectors are orthogonal precisely when a^\top b = 0. Perpendicularity, the most geometric idea in the subject, reduces to one arithmetic condition: a dot product that vanishes. Hold onto that; the whole essay turns on it.

Projection onto a line

Start with the smallest possible subspace: a single line through the origin, all the multiples of one vector a. We have another vector b floating somewhere off the line, and we want its shadow — the point \hat b on the line that sits closest to b.

Because \hat b is on the line, it's some scalar multiple of a: write \hat b = x\,a, and we just need the right scalar x. Here's where orthogonality earns its keep. The closest point is the foot of the perpendicular, which means the leftover b - \hat b — the residual — must be perpendicular to the line, hence to a itself:

a^\top (b - x\,a) = 0 \quad\Longrightarrow\quad a^\top b - x\,(a^\top a) = 0 \quad\Longrightarrow\quad x = \frac{a^\top b}{a^\top a}.

So the projection of b onto the line through a is

\boxed{\;\hat b = \frac{a^\top b}{a^\top a}\,a\;}

Look at what this says. The numerator a^\top b measures how much b agrees with a; the denominator a^\top a = \lVert a \rVert^2 normalizes away the length of a so that scaling a doesn't change where the shadow lands. If a happens to be a unit vector, with \lVert a \rVert = 1, this collapses to the clean \hat b = (a^\top b)\,a — the component of b along a, no bookkeeping required. That's the first hint of why we'll soon want our basis vectors to have length one.

Projection onto a subspace

A line was the warm-up. The real problem: project b onto a whole subspace — the column space of a matrix A whose columns are the directions we're allowed to combine. Now \hat b is a combination of all the columns, \hat b = A\hat x for some coefficient vector \hat x, and the geometry is identical to the line case, only the residual has to be perpendicular to every column at once.

"Perpendicular to every column of A" is exactly what A^\top tests, because each row of A^\top is a column of A, and multiplying by A^\top takes the dot product against each one. So the perpendicularity condition is

A^\top (b - A\hat x) = 0 \quad\Longrightarrow\quad A^\top A\,\hat x = A^\top b.

These are the normal equations — "normal" in its old geometric sense of perpendicular, not ordinary. When the columns of A are independent, A^\top A is invertible (that's the payoff from the four-subspaces essay: independent columns mean a trivial null space, and nothing for A^\top A to crush), so \hat x = (A^\top A)^{-1} A^\top b and the projection itself is

\hat b = A\hat x = A(A^\top A)^{-1} A^\top\, b = P\,b, \qquad \boxed{\;P = A(A^\top A)^{-1} A^\top\;}

P is the projection matrix: feed it any b and it returns the shadow of b on the column space. It has two properties that are really just geometry wearing algebra's clothes. It's symmetric, P^\top = P. And it's idempotent, P^2 = P — project once and you're already in the subspace, so projecting again does nothing; the shadow of a shadow is the shadow. You can check P^2 = P by multiplying it out and watching the inner (A^\top A)^{-1}(A^\top A) collapse to the identity, but the picture tells you the answer before the algebra does.

The residual is always perpendicular

Every projection splits b cleanly into two pieces that meet at a right angle:

b = \underbrace{\hat b}_{\text{in } C(A)} + \underbrace{(b - \hat b)}_{\perp\, C(A)}.

The shadow \hat b lives inside the subspace; the residual b - \hat b lives in its orthogonal complement — for the column space of A, that's the left null space N(A^\top), the other side of the mirror from the four-subspaces picture. The two pieces are perpendicular by construction, which is why the projection is not just a decent point but the closest one. The Pythagorean theorem makes this precise: for any other point A x in the subspace,

\lVert b - A x \rVert^2 = \lVert b - \hat b \rVert^2 + \lVert \hat b - A x \rVert^2 \;\ge\; \lVert b - \hat b \rVert^2,

because \hat b - Ax lies in the subspace while b - \hat b is perpendicular to it, so the two legs form a right triangle with b - Ax as hypotenuse. Moving away from \hat b inside the subspace can only lengthen the distance to b. That single inequality is the whole justification for calling \hat b the best approximation — and it's precisely the statement that least squares minimizes, arrived at by pure geometry instead of by setting a derivative to zero.

Gram–Schmidt: manufacturing perpendicular directions

Notice how much simpler life got when a was a unit vector, and imagine how much simpler the whole subspace story would be if the columns of A were not just independent but mutually perpendicular and unit length — an orthonormal set. Then A^\top A would be the identity, the dreaded inverse would vanish, and projection would be nothing but a sum of one-dimensional shadows: \hat b = \sum_i (q_i^\top b)\,q_i, one term per basis vector. So the practical question is: given any independent columns, can we manufacture an orthonormal basis for the same subspace?

We can, and the recipe is Gram–Schmidt — a procedure that traces to Jørgen Pedersen Gram's work of 1883 and Erhard Schmidt's cleaner formulation in 1907. The idea is disarmingly simple, and it's just projection used as a subtraction tool. Take the vectors a_1, a_2, \ldots one at a time. Keep the first as your first direction. For each new vector, project it onto everything you've already accepted and subtract that shadow off — what remains is, by the residual argument above, perpendicular to all of them. Then rescale to length one:

q_1 = \frac{a_1}{\lVert a_1 \rVert}, \qquad v_2 = a_2 - (q_1^\top a_2)\,q_1, \qquad q_2 = \frac{v_2}{\lVert v_2 \rVert},

and in general you subtract from a_k its shadow on each earlier q_i before normalizing. Every step removes the part that overlaps with directions you already have, leaving only genuinely new, perpendicular information. Strip away the parts that agree, keep the part that's at right angles — Gram–Schmidt is that instinct written as a loop.

QR: the bookkeeping that falls out

Here's the quiet bonus. When you run Gram–Schmidt, every original column a_k is some combination of the orthonormal vectors q_1, \ldots, q_k you built up to that point — never any later ones, because those didn't exist yet. Collect the q's as columns of a matrix Q and the combination-coefficients as an upper-triangular matrix R, and the whole process is one clean factorization:

\boxed{\;A = QR\;}

with Q having orthonormal columns (Q^\top Q = I) and R upper triangular. This is the QR decomposition, and it turns the normal equations into something a computer can solve without ever forming the numerically nasty A^\top A: substitute A = QR and the equation A^\top A\hat x = A^\top b collapses, via Q^\top Q = I, to R\hat x = Q^\top b — a triangular system you solve by back-substitution in one sweep. The projection matrix simplifies too, to the tidy P = QQ^\top. Orthonormal bases don't just look elegant; they're what makes the geometry computationally honest.

Where this goes

Projection is the hinge of the whole applied story. The moment your target won't sit in the reachable subspace — which, with real, noisy data, is essentially always — you stop demanding an exact solution and start asking for the closest one, and the closest one is a shadow with a perpendicular residual. Make that concrete with a cloud of data points and a line, and you have least squares; wrap it in a probability model with noise and uncertainty, and you have linear regression. Both are this essay's geometry with statistics bolted on.

But the fastest way to feel it is to move it with your own hand. The orthogonality-projection playground fixes a subspace and lets you drag the vector b wherever you like. Watch the shadow \hat b slide along the subspace to stay directly "beneath" b, watch the gold residual stretch and shrink while stubbornly holding its right angle, and watch the numbers update as you go. Drag b until it lands on the subspace and the residual vanishes — that's the solvable case from the four-subspaces essay, the shadow and the object finally coinciding. It's one thing to read that the residual is always perpendicular; it's another to try, and fail, to make it point any other way.