Adam: the optimizer that keeps its own running notes

Gradient descent is the honest workhorse of machine learning: compute the slope, take a step downhill, repeat. It's in gradient descent in full, and it works — until it doesn't. Drop it into a long, narrow valley and it starts to sputter, bouncing off the steep walls while barely creeping along the gentle floor. Feed it noisy or lopsided gradients and it either crawls or overshoots, depending on the one learning rate you were forced to pick for every parameter at once. The trouble is that plain descent has no memory and no sense of scale. Each step is computed from scratch, blind to everything that came before.

Adam — short for adaptive moment estimation, introduced by Kingma and Ba in 2014 — is what you get when you give the optimizer a memory and a ruler. It has become the default first thing to try on almost any deep network, and its popularity is not an accident: it combines two older good ideas so cleanly that the combination feels inevitable. Let's climb to it one rung at a time, because every piece of Adam is a fix for a specific failure of the rung below.

The ladder

Rung one: plain SGD. Stochastic gradient descent updates the parameters by walking directly against the gradient:

\theta_t = \theta_{t-1} - \eta\, g_t,

where g_t is the gradient at step t and \eta is the learning rate. Simple, but stubborn. In a ravine — a loss surface that's steep across and shallow along — the gradient points mostly across the valley, so SGD zig-zags between the walls and inches down the axis it actually wants to travel. It has no way to notice that it keeps being sent back and forth in the same two directions.

Rung two: momentum. So give it a memory of its recent direction. Instead of stepping along the raw gradient, keep an exponential moving average of the gradients and step along that:

m_t = \beta_1 m_{t-1} + (1-\beta_1)\, g_t, \qquad \theta_t = \theta_{t-1} - \eta\, m_t.

An exponential moving average (EMA) is just a running weighted mean where old information decays geometrically — with \beta_1 = 0.9, roughly the last ten gradients still have a say, and anything older has faded to nothing. In a ravine the across-valley components keep flipping sign and cancel in the average, while the along-valley component points the same way every step and accumulates. Momentum builds speed in the consistent direction and damps the oscillation, exactly like a heavy ball that stops rattling off the walls and starts rolling down the trough.

Rung three: RMSProp. Momentum fixes direction but not scale. One learning rate still has to serve every parameter, even when some coordinates see huge gradients and others see tiny ones. RMSProp keeps a second EMA — this time of the squared gradients — and uses it to normalize each coordinate's step:

v_t = \beta_2 v_{t-1} + (1-\beta_2)\, g_t^2, \qquad \theta_t = \theta_{t-1} - \frac{\eta}{\sqrt{v_t}+\varepsilon}\, g_t.

Here g_t^2 is elementwise, so v_t tracks the recent typical size of the gradient in each direction separately. Dividing by \sqrt{v_t} shrinks the step where gradients have been large and volatile, and stretches it where they've been small and steady. Every parameter effectively gets its own learning rate, tuned on the fly. The little \varepsilon (something like 10^{-8}) is just a bouncer at the door, there to stop a division by zero when a gradient has been quiet.

Rung four: Adam. Momentum remembers which way; RMSProp remembers how big. Adam is the obvious question nobody had to be clever to ask — why not both? Keep the first-moment EMA for direction and the second-moment EMA for scale, and use each for what it's good at.

Two moments

Adam's state is those two running averages, computed at every step from the current gradient g_t:

m_t = \beta_1 m_{t-1} + (1-\beta_1)\, g_t,

v_t = \beta_2 v_{t-1} + (1-\beta_2)\, g_t^2.

The names come from statistics. The first moment of a distribution is its mean, and m_t is an estimate of the mean of the gradient — the direction it's been pointing on average. The second (raw) moment is the mean of the square, and v_t estimates the mean of the squared gradient — its typical magnitude, uncentered. So Adam is quietly doing statistics on the stream of gradients it sees: what's your average, and what's your scale? The usual defaults are \beta_1 = 0.9 (the direction memory is short-ish) and \beta_2 = 0.999 (the scale memory is much longer, so per-parameter step sizes change slowly and don't jitter).

Why the early steps lie

There's a subtle bug in those two EMAs, and it's worth seeing rather than taking on faith. We initialize m_0 = 0 and v_0 = 0 — we have no history to start from, so we start from nothing. But "nothing" is a number, and it drags the average toward zero for a while.

Watch the first step. With m_0 = 0,

m_1 = \beta_1 \cdot 0 + (1-\beta_1)\, g_1 = (1-\beta_1)\, g_1 = 0.1\, g_1

when \beta_1 = 0.9. The gradient was g_1, but our "average gradient" reads a tenth of it. The estimate is biased toward zero, badly, precisely when we've seen the least data — and v_t is worse, dragged down by a factor of 1-\beta_2 = 0.001 on its first step. Left uncorrected, Adam would tiptoe through the opening steps with a first moment that's far too small and a second moment that's far too small under the square root, a confusing mix that makes the early trajectory unreliable.

The fix is arithmetic, not hand-waving. If the true mean of the gradient were a constant g, unrolling the recursion gives m_t = (1-\beta_1^t)\, g — the estimate is the truth scaled down by exactly 1 - \beta_1^t. So divide it back out:

\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t}.

At t=1 this multiplies m_1 by 1/(1-\beta_1) = 10, undoing the shrinkage exactly and recovering g_1. As t grows, \beta_1^t \to 0, the correction factor \to 1, and it quietly switches itself off once the averages have enough history to stand on their own. It's a warm-up baked into the algebra: loud when there's no data, silent once there is.

The update

Put the corrected moments together and Adam's step is:

\theta_t = \theta_{t-1} - \eta\, \frac{\hat m_t}{\sqrt{\hat v_t}+\varepsilon}.

Read it as a sentence. The numerator \hat m_t is the smoothed direction — where have we been heading. The denominator \sqrt{\hat v_t} is the smoothed magnitude — how big have the gradients been in this coordinate. Their ratio is roughly a signal-to-noise measure: when a coordinate's gradient points consistently one way, \hat m_t is large relative to \sqrt{\hat v_t} and Adam takes a confident step; when a coordinate is just twitching noisily, the two are comparable and the step shrinks. The learning rate \eta sets the overall scale, but each parameter gets its step individually sized. That's the whole algorithm — two EMAs, one bias correction, one division.

When Adam helps, and when it doesn't

Adam earns its keep exactly where plain SGD struggles. In ravines the momentum term rolls smoothly along the floor instead of clanging off the walls. With sparse gradients — a parameter that only occasionally sees a nonzero signal, common in embeddings and NLP — the per-parameter scaling means that rare-but-real gradient still produces a meaningful step instead of being drowned out by a global learning rate tuned for busier parameters. And with noisy gradients the two levels of averaging smooth out the jitter that would send SGD staggering. For getting a network to train at all, quickly and without much tuning, Adam is a superb default.

But be honest about the trade. There's a well-known and slightly awkward finding that on many problems plain SGD with momentum, given enough patience and a good learning-rate schedule, generalizes better — it finds minima that perform a shade better on unseen data, even when Adam reached a lower training loss faster. The adaptive per-parameter scaling that makes Adam converge quickly may steer it toward minima that fit the training set a little too snugly. So the rule of thumb most practitioners land on: reach for Adam to move fast and to tame ill-conditioned or sparse problems, and consider well-tuned SGD when squeezing out the last drops of test-set performance is what matters. No free lunch — just a well-stocked one.

The fastest way to feel all of this is to watch it. The Adam playground drops all four optimizers — SGD, momentum, RMSProp, and Adam — onto the same loss surface from the same starting point and races them. Switch to the ravine and watch SGD zig-zag while Adam cuts to the floor; try the saddle and see who escapes; drag the learning rate and the betas and watch each trajectory react. It's the difference between reading about a heavy ball rolling down a canyon and rolling one yourself.