Experiments / live

Adam playground

Four optimizers, one landscape, one starting point. From a fixed start, plain SGD, momentum, RMSProp and Adam each run for a fixed number of steps and leave a colored trail across the contours. Switch the surface to a ravine and watch SGD zig-zag off the walls while Adam cuts to the floor; drag η and the βs and watch every trail react. The white dot is the minimum. The full story — the ladder from SGD to Adam and why the early steps need a bias correction — is in the essay.

loss surface — trajectories

surface

η 0.050 β1 0.90 β2 0.9990 ε 1e-8

SGD Momentum RMSProp Adam

This playground is interactive and needs JavaScript. Here's the idea as a static picture — four optimizers descend the same ravine from the same start; SGD zig-zags off the walls while Adam cuts almost straight to the floor: