Experiments / live
Softmax playground
Drag the ink bars on the left to set the raw logits z — unbounded scores, no rules. The steel bars on the right are what softmax makes of them: a probability distribution that always sums to 1. Turn the temperature T up and the distribution flattens toward uniform; turn it down and it sharpens toward a one-hot spike on the winner. The logits never move — only how loudly softmax listens to their differences. The full story — shift invariance, the sigmoid connection, and the clean gradient — is in the essay.
logits z — drag me
softmax(z / T) — sums to 1
the log-sum-exp trick
Naive softmax exponentiates the logits directly, so a big logit sends e^z to Infinity and the whole distribution becomes NaN. The stabilized version subtracts the largest logit first (softmax(z) = softmax(z − max z), so the answer is unchanged) and stays finite. Same math, one column survives:
naive: exp(z) / Σ exp(z)
stable: exp(z − max z) / Σ exp(z − max z)