Eigenvectors and the spectral theorem

We met a matrix as a verb: a thing that grabs the plane and bends it, sending most vectors off in some new direction, turned and stretched at once. Watch that bending happen and you can't help noticing that not every vector is treated alike. Point an arrow one way and the map swings it hard to the side; point it another way and — nothing turns. The arrow comes out longer or shorter, maybe flipped end to end, but lying along the exact same line it started on. The map left its direction untouched and only messed with its length. Those special, un-turned directions are the eigenvectors, and the amount each one stretches by is its eigenvalue. They are the closest thing a matrix has to a personality: strip away everything the map does to everything else, and what's left is a short list of favorite directions and how hard it pulls along each.

That instinct — that a transformation has privileged directions where it acts by simple scaling — runs right back through the nineteenth century. Cauchy, working on the principal axes of quadratic surfaces in the 1820s, was already computing what we'd now call eigenvalues and proving that for the symmetric case they come out real. The names came later — Hilbert attached the German eigen, "own" or "characteristic," to the whole circle of ideas around 1904 — but the geometry was there from the start: every well-behaved map has axes of its own, and along those axes it does nothing more exotic than multiply. This essay is about finding them, what they buy you, and the beautiful thing that happens when the matrix is symmetric.

The eigen-equation

Write the wish down and it's almost embarrassingly simple. A nonzero vector v is an eigenvector of the square matrix A if applying A only scales it:

Av = \lambda v.

The scalar \lambda is the eigenvalue — the stretch factor along that direction. That's the entire definition. Av would normally be a brand-new vector pointing somewhere else; the equation demands that it instead land back on the line through v, just farther out (\lambda > 1), pulled in (0 < \lambda < 1), flipped (\lambda < 0), or annihilated (\lambda = 0, which — no accident — is exactly the null-space direction of a singular matrix). Everything else in the plane gets rotated to some degree as the map bends it; the eigen-directions are the ones the rotation forgets.

The catch is that v appears on both sides, so we can't just read the answer off. Rearrange:

Av - \lambda v = 0 \quad\Longrightarrow\quad (A - \lambda I)v = 0,

slipping in the identity I so the subtraction makes sense between two matrices. Now it says something sharp. We need a nonzero v that the matrix A - \lambda I sends to zero — a vector living in its null space. But a matrix only has a nontrivial null space when it's singular, when it collapses space flat, and we already know the one-number test for that from the matrix-as-map essay: its determinant must vanish.

The characteristic polynomial

So the eigenvalues are exactly the numbers \lambda that make A - \lambda I singular:

\det(A - \lambda I) = 0.

Expand that determinant and, for an n \times n matrix, out drops a degree-n polynomial in \lambda — the characteristic polynomial. Its roots are the eigenvalues. For the 2\times2 case that the playground lives in, it's worth grinding all the way to the end, because the answer is clean and closed-form. With

A = \begin{bmatrix} a & b \\ c & d \end{bmatrix}, \qquad \det(A - \lambda I) = (a-\lambda)(d-\lambda) - bc = \lambda^2 - (a+d)\lambda + (ad - bc).

The two coefficients are old friends: a + d is the trace (call it \tau) and ad - bc is the determinant (\delta). The quadratic formula then hands you both eigenvalues at once:

\lambda = \frac{\tau}{2} \pm \sqrt{\left(\frac{\tau}{2}\right)^{2} - \delta}.

Read the discriminant like a weather report. When (\tau/2)^2 \ge \delta you get two real eigenvalues (equal, if it's exactly zero) — two genuine directions the map merely scales. When (\tau/2)^2 < \delta the square root goes imaginary: there is no real eigenline at all. Geometrically the map is doing an honest rotation, spinning every direction to a new one, so of course no arrow survives pointing the same way. A rotation has no favorite direction in the real plane, and the algebra tells you so by refusing to give real roots. (The playground shows "complex" here rather than pretending — turning a rotation loose and demanding a fixed direction is asking for something that isn't there.)

Diagonalization: changing into the map's own coordinates

Suppose an n \times n matrix has n independent eigenvectors — enough to build a basis out of them. Stack those eigenvectors as the columns of a matrix V, and put the eigenvalues down the diagonal of an otherwise-empty matrix \Lambda. The eigen-equations A v_i = \lambda_i v_i, all n of them, bundle into one clean statement, AV = V\Lambda, and since V is invertible,

\boxed{\;A = V \Lambda V^{-1}\;}

This is diagonalization, and the way to hear it is as a change of language. V^{-1} rewrites any vector in the eigenvector basis — in the map's own coordinate system. \Lambda, being diagonal, then does the only thing a diagonal matrix can: it scales each eigen-coordinate independently, by its eigenvalue, no mixing. Then V translates the result back into ordinary coordinates. A matrix that looked like a tangle of shearing and rotating is revealed to be, in the right frame, nothing but n separate stretches happening along n independent axes. All the apparent complication was an artifact of describing the map in the wrong coordinates.

Why powers become trivial

Here is where diagonalization stops being pretty and starts paying rent. Suppose you need A^k — the map applied k times over, the thing that governs every process that iterates, from Markov chains to the repeated multiplications inside a recurrent model. Multiply the factored form by itself and watch the middle collapse:

A^2 = (V\Lambda V^{-1})(V\Lambda V^{-1}) = V\Lambda \underbrace{V^{-1}V}_{I}\Lambda V^{-1} = V\Lambda^2 V^{-1},

and the same telescoping runs all the way up:

A^k = V\,\Lambda^{k}\,V^{-1}.

Every interior V^{-1}V cancels to the identity, leaving a single power of the diagonal matrix in the middle — and a diagonal matrix raised to the k-th power is just each diagonal entry raised to the k-th, \Lambda^k = \operatorname{diag}(\lambda_1^k, \ldots, \lambda_n^k). No matrix multiplication at all, only n scalar powers. A calculation that naively costs you k full matrix products becomes three matrices and a handful of exponentials. And it tells you the long-run behavior for free: whichever |\lambda_i| is largest dominates the sum as k grows, so the whole system, whatever it looked like at the start, ends up lining itself up along the eigenvector of the biggest eigenvalue. Hold onto that — it's about to become an algorithm.

The spectral theorem: symmetric matrices get perfect axes

Diagonalization is a lovely deal when you can get it, but in general the eigenvectors need not be perpendicular, V^{-1} can be a numerical menace, and — as we just saw — the eigenvalues can go complex and the whole thing can fail to diagonalize over the reals at all. Then you meet a symmetric matrix, one with A = A^\top, and every one of those worries evaporates at once. This is the spectral theorem, and it is one of the crown jewels of the subject:

If A is real and symmetric, then all its eigenvalues are real, and its eigenvectors can be chosen to be orthonormal — mutually perpendicular and unit length.

An orthonormal eigenbasis means the matrix V we stack them into isn't just any invertible matrix; it's an orthogonal matrix Q, one whose inverse is simply its transpose, Q^{-1} = Q^\top (that's exactly what orthonormal columns give you). So the awkward V^{-1} in the diagonalization is replaced by a free transpose, and the factorization sharpens into

\boxed{\;A = Q\,\Lambda\,Q^\top\;}

with Q^\top Q = I and \Lambda real and diagonal. Geometrically this is the cleanest possible picture of a map: rotate into a perpendicular set of axes (Q^\top), stretch along each by a real factor (\Lambda), rotate back (Q). No shear, no skew, no complex bookkeeping — a symmetric matrix is pure stretching along orthogonal directions, full stop. Cauchy's principal-axis theorem, dressed in modern notation. This is not a niche case, either: symmetric matrices are everywhere the moment you form A^\top A, a covariance matrix, or the Hessian of a smooth function, which is precisely why the spectral theorem is the hinge that positive-definite matrices, the singular value decomposition, and principal component analysis all turn on.

Power iteration: chasing the dominant eigenvector

We noticed above that the largest-magnitude eigenvalue takes over as you apply a matrix again and again. Turn that observation around and it becomes the simplest eigenvector algorithm there is — power iteration. Start with almost any vector v_0 and just keep hitting it with A, renormalizing each time so it doesn't blow up or vanish:

v_{k+1} = \frac{A v_k}{\lVert A v_k \rVert}.

To see why it works, write the starting vector in the eigenbasis, v_0 = c_1 v_1 + c_2 v_2 + \cdots. Each application of A multiplies the i-th piece by \lambda_i, so after k steps the piece along v_1 (the dominant direction, |\lambda_1| largest) has grown by \lambda_1^k while every other piece grows only by its own smaller \lambda_i^k. Factor out \lambda_1^k and the rival terms carry factors (\lambda_i/\lambda_1)^k, each a number smaller than one in magnitude being driven toward zero. The dominant direction wins by attrition. Normalize away the overall scale and v_k homes in on the top eigenvector, no characteristic polynomial required — which is why a version of it, iterated over the whole web's link matrix, is the seed of PageRank. It converges fast when |\lambda_1| comfortably beats |\lambda_2| and slowly when they're close, and that race is exactly what you can watch unfold by tapping the step button below.

Where this goes

Eigenvectors are the moment a matrix stops being a wall of numbers and becomes a set of independent stretches along its own natural axes. That single reframing is the load-bearing beam under a startling amount of what follows. Ask when all those stretch factors are positive — when the map never flips any direction and the associated quadratic form stays uphill everywhere — and you have positive-definite matrices, the shape of every well-behaved optimization landscape. Ask what to do when the matrix isn't square or symmetric and has no eigen-decomposition of its own, and the fix is to borrow the spectral theorem for the symmetric A^\top A instead: that detour is the singular value decomposition. Point the whole apparatus at a covariance matrix and its top eigenvectors become the directions your data actually varies along — the essence of principal component analysis.

But the fastest way to believe any of it is to hunt the directions yourself. The eigenvectors playground lets you set the four entries of a 2\times2 matrix and sweeps a probe vector around the unit circle, drawing each candidate beside its image Av; the directions that don't rotate — where the image lies flat along the original — light up as the eigen-directions, and the readout shows the eigenvalues (or tells you honestly when the map is a rotation with none). Then hit the step button and watch power iteration take over: one v \leftarrow Av/\lVert Av\rVert at a time, a lost little arrow swinging around and settling, step by step, onto the dominant eigenvector. It's one thing to read that a matrix has favorite directions. It's another to spin an arrow around until you find them.