Gradient descent: the art of rolling downhill

There are two ways to find the bottom of a valley. You can survey the entire terrain, write down its equation, and solve for the lowest point in one shot — which is exactly what least squares does when it sets a gradient to zero and reads off the normal equations. Or you can stand where you are, feel which way is downhill, take a step, and repeat until your feet stop dropping. The first way is elegant and exact. The second way is what you actually do when the valley is too big to survey, or too crooked to have a closed-form bottom at all.

Almost every model worth training lives in that second world. Millions of parameters, a loss surface you can never write down in full, no tidy matrix to invert. And yet the recipe stays absurdly simple, which is the whole charm of it. Feel the slope. Step against it. Repeat.

The idea: follow the negative gradient

Suppose you're minimizing a loss L(\theta) over a parameter vector \theta. The gradient \nabla L(\theta) is the vector of partial derivatives — it points in the direction of steepest ascent, the way the loss climbs fastest. We want to go down, so we walk the other way. That's the entire algorithm:

\theta \leftarrow \theta - \eta\,\nabla L(\theta).

Read it left to right. Start somewhere. Compute the slope at your feet. Take a step of size \eta in the direction that decreases the loss most steeply. Land at a new \theta, and do it again. The number \eta — the learning rate — sets how big each step is.

Notice something the picture makes obvious. The steps get smaller on their own as you near the bottom, even though \eta never changes. That's because the gradient itself shrinks toward zero as the slope flattens out — near the minimum the ground is nearly level, so \nabla L is tiny, so \eta\,\nabla L is tiny. Gradient descent naturally decelerates into a minimum. It doesn't need to be told it's arriving; the terrain tells it.

And here's the connection worth holding onto. That bowl in the figure is the same loss surface least squares minimizes — the sum of squared residuals L(\beta) = \lVert X\beta - y\rVert^2, a perfect quadratic that curves upward in every direction. Least squares finds its floor by solving X^\top X\beta = X^\top y in one move. Gradient descent finds the identical floor by rolling into it. Same minimum, two roads: one you compute, one you walk.

The learning rate is the whole ballgame

If the update rule is trivial, why does anyone ever fail at this? Because of \eta. It is a single positive number and it is responsible for nearly every training disaster you'll ever meet.

Set \eta too small and each step barely moves you. You do converge — the direction is right, so you're always heading downhill — but you crawl. Thousands of iterations to cross a distance a bolder step would clear in ten. On a real model that's the difference between training overnight and training for a month.

Set \eta too large and something worse happens. You feel the downhill direction correctly, but you step so far that you sail past the bottom and land on the opposite wall of the valley — higher than where you started. Now the slope there points back the other way, so your next giant step overshoots again, even farther out. The iterates bounce from wall to wall with growing amplitude, and the loss doesn't decrease — it explodes. This is divergence, and it's the failure mode that turns a loss curve into a spray of NaNs.

Picture it on a 1D parabola L(\theta) = \tfrac{1}{2}k\theta^2. The gradient is L'(\theta) = k\theta, so the update is

\theta \leftarrow \theta - \eta k\theta = (1 - \eta k)\,\theta.

Every step multiplies your position by the fixed factor (1 - \eta k). The behaviour hangs entirely on that number. If 0 < \eta k < 1, the factor is a positive fraction and \theta shrinks smoothly toward zero. If 1 < \eta k < 2, the factor is negative but smaller than one in size, so \theta flips sign each step while still shrinking — you oscillate across the minimum but close in. And if \eta k > 2, the factor exceeds one in magnitude, so \theta grows every step: you're thrown outward, forever. One knob, three completely different fates, and the boundary is the crisp condition \eta < 2/k.

Convergence: when rolling downhill actually works

The parabola isn't just a toy — it's the local picture of any smooth loss near a minimum, and it hands us the general story. The parameter k was the curvature of the bowl; in many dimensions the role of k is played by the largest curvature of the surface, formally the biggest eigenvalue L of the Hessian (the "L-smoothness" constant). The same threshold reappears: for a smooth loss, a fixed step size \eta < 2/L keeps you from overshooting into divergence, and \eta \le 1/L gives you a clean, monotone march to the bottom.

But "the bottom" is only guaranteed to be the bottom when the surface is convex — bowl-shaped, curving upward everywhere, with no false floors. For a convex, smooth L, gradient descent provably converges to the global minimum, and the loss gap shrinks like O(1/t) with the number of steps t (faster, geometrically, if the bowl is also strongly convex — curved up by at least some fixed amount in every direction). This is the happy case, and it's exactly the least-squares bowl: convex by construction, since its Hessian 2X^\top X is positive semidefinite. Roll downhill on that surface with a sane step size and you will reach the least-squares solution. The normal equations just get you there in one line instead.

There's a subtlety worth a sentence. A step size small enough to guarantee convergence can be painfully slow, and a step size fast enough to be quick can flirt with overshoot. Living in that gap — big steps early when you're far out, smaller steps later as you home in — is the entire motivation for learning-rate schedules and adaptive methods. But the fixed-\eta intuition is the foundation: too small crawls, too big diverges, and there's a Goldilocks band in between whose width is set by the curvature of your loss.

Convex versus non-convex: where the trouble lives

Convex losses are the comfortable exception. The surfaces that real neural networks ride on are non-convex — crumpled landscapes with hills, basins, ridges, and plateaus — and there the guarantee evaporates. Gradient descent only ever feels the slope right where it stands; it has no map. So it can only promise to find a point where the slope is zero. It cannot promise that point is the lowest one.

Two things can trap it. A local minimum is a genuine basin — lower than everything nearby, but not the deepest valley on the map. Roll into one and the gradient vanishes; you stop, contentedly, at a floor that isn't the floor. A saddle point is sneakier: the surface curves up in some directions and down in others, like a mountain pass or a Pringle. The gradient is zero right at the saddle, so it looks like a resting place, but it isn't a minimum at all — there's a downhill escape route, if only you're not sitting exactly on the ridge. In high dimensions saddles vastly outnumber local minima, and much of the craft of modern optimization is really about not getting stuck loitering on the flat plateaus around them.

None of this makes the method useless — far from it; it trains essentially every deep model in existence. It just means the clean convex guarantees become empirical good behaviour, helped along by randomness, momentum, and a great deal of accumulated folklore. The best cure for hand-waving here is to watch it happen: the gradient descent playground lets you drop a starting point on a bowl, a ravine, and a saddle in turn, drag the learning rate up until the 1D iterates visibly explode, and see the 2D trajectory zig-zag down a narrow valley or slide off a saddle. Crank \eta past the divergence threshold and watch the dots march off to infinity — it's the fastest way to feel why this one number keeps machine-learning engineers up at night.

A teaser: giving the ball some memory

Plain gradient descent has a frustrating habit in long, narrow valleys — the "ravine" you can try in the playground. Across the valley the walls are steep, so the gradient there is large; along the valley toward the true minimum the floor is nearly flat, so the gradient there is tiny. The result is a maddening zig-zag: the steps bounce back and forth across the narrow direction while barely creeping along the direction you actually care about.

The fix is intuitive if you think of the ball physically: give it momentum. Instead of stepping purely by the current gradient, accumulate a running velocity that remembers past steps:

v \leftarrow \gamma v + \eta\,\nabla L(\theta), \qquad \theta \leftarrow \theta - v.

Now the back-and-forth zig-zags across the valley tend to cancel — they keep reversing sign — while the consistent nudge along the valley accumulates and builds speed. The ball stops rattling and starts rolling. Add per-parameter step sizes that adapt to how steep each direction has historically been, blend in the momentum, and you arrive at the optimizer that trains most of modern deep learning: Adam. But every one of those refinements is a patch on the same beating heart — feel the slope, step against it, repeat. Learn to see that heart clearly and the rest is variations on a theme.

The playground is waiting: pick a surface, find the learning rate that's fast but doesn't blow up, and watch a very simple rule do a surprising amount of work.