Cross entropy: the price of believing the wrong distribution

Ask a machine to sort photos into "cat" and "dog" and at some point it has to be graded. It hands you a probability — "80% cat" — and reality hands back the truth — "it was a cat." How wrong was the guess, exactly? Not "wrong" as a yes/no verdict, but as a number you can differentiate and roll downhill. Cross entropy is the answer nearly everyone uses, and the lovely thing is that it isn't a convenient hack. It's the honest cost, in bits, of describing the world with the wrong probabilities — a quantity information theory had already worked out decades before anyone trained a neural network with it. Let's build it from the ground up.

Entropy: expected surprise

Start with a single distribution p over outcomes — say the probabilities of tomorrow's weather. When an event with probability p_i actually happens, how surprised should you be? A sure thing (p_i = 1) is no surprise at all; a one-in-a-million event is a jolt. The function that behaves this way is -\log p_i: it's zero when p_i = 1 and shoots up as p_i \to 0. Call that the surprisal of outcome i.

Entropy is just the average surprise, weighted by how often each outcome occurs:

H(p) = -\sum_i p_i \log p_i.

It's the expected surprisal of a draw from p — and, read through Shannon's lens, the average number of bits (if \log is base 2) you need to encode outcomes drawn from p when you use an optimal code. A fair coin has H = 1 bit: maximally uncertain, one bit per flip. A weighted coin that lands heads 99% of the time has entropy near zero, because you're almost never surprised, so you almost never need to spend bits saying what happened. Entropy is uncertainty made quantitative.

KL divergence: the cost of the wrong code

Now suppose reality runs on p, but you don't know p — you believe some other distribution q, and you build your code as if q were true. You'll still get the messages across, but you'll waste bits, because you've optimized for the wrong world. That waste has a name: the Kullback–Leibler divergence,

D_{KL}(p \,\|\, q) = \sum_i p_i \log \frac{p_i}{q_i}.

Read it as an expectation over the real distribution p of the log-ratio between what you believed and what was true. Two properties earn their keep. First, it is always non-negative, D_{KL}(p \,\|\, q) \ge 0, with equality exactly when q = p — you can't do better than the truth, only worse. (This is Gibbs' inequality; it follows from the concavity of \log via Jensen's inequality.) Second, it is not symmetric: D_{KL}(p \,\|\, q) \ne D_{KL}(q \,\|\, p) in general. The cost of believing q when the world is p is a different number from the reverse. So KL is a directed discrepancy, not a distance — it doesn't obey the triangle inequality and it isn't a metric, however tempting it is to call it one.

Cross entropy: entropy plus the penalty

Cross entropy is what you get when you actually pay both bills at once — the irreducible surprise of the world plus the penalty for describing it with the wrong distribution:

H(p, q) = -\sum_i p_i \log q_i.

It's the expected surprisal again, but with a twist: the outcomes still occur with the true frequencies p_i, while the surprisal -\log q_i is computed against your believed distribution q. The identity that ties the whole essay together drops out with one line of algebra. Split the log:

D_{KL}(p \,\|\, q) = \sum_i p_i \log \frac{p_i}{q_i} = \sum_i p_i \log p_i - \sum_i p_i \log q_i = -H(p) + H(p, q).

Rearrange and you have it:

\boxed{\,H(p, q) = H(p) + D_{KL}(p \,\|\, q)\,}

Cross entropy is the entropy of the truth plus the KL penalty for using the wrong distribution. Since H(p) doesn't depend on your prediction q at all — it's a fixed property of the world — minimizing cross entropy over q is exactly minimizing the KL divergence. And because D_{KL} \ge 0 with equality only at q = p, the cross entropy bottoms out precisely when your predicted distribution matches the truth. That's the whole justification for using it as a loss: drive H(p, q) down and you are provably dragging q toward p.

As a loss: it's negative log-likelihood in disguise

Here's where information theory and statistics turn out to be the same conversation. In classification, the "true" distribution p for a single labelled example is one-hot: all the probability mass sits on the correct class c, so p_c = 1 and every other p_i = 0. Plug that into the cross-entropy sum and every term vanishes except one:

H(p, q) = -\sum_i p_i \log q_i = -\log q_c.

The loss for one example is just the negative log of the probability your model assigned to the correct class. Minimizing it means pushing q_c toward 1 — maximizing the log-probability of the right answer. Sum over a dataset and minimizing total cross entropy is identical to maximum likelihood estimation: it's the negative log-likelihood, the same principle that gave us the least-squares line under Gaussian noise, now wearing categorical clothes.

Two everyday special cases. With K classes and a one-hot target, the categorical cross entropy is -\log q_c as above. With two classes you can track a single probability \hat y = q_1 for "positive" and 1 - \hat y for "negative," and with true label y \in \{0, 1\} the binary cross entropy is

L = -\big[\,y \log \hat y + (1 - y)\log(1 - \hat y)\,\big].

When y = 1 only the first term survives and the loss is -\log \hat y; when y = 0 only the second survives and it's -\log(1 - \hat y). Either way, growing confidence in the wrong direction sends the loss toward infinity — the vertical wall in the figure. That savagery is a feature: it makes the model deeply reluctant to be confidently wrong.

Why it loves softmax and sigmoid

The last piece is why this particular loss shows up glued to particular output layers. A classifier produces raw scores — logits z_i — and squashes them into probabilities. For many classes that's the softmax, q_i = e^{z_i} / \sum_j e^{z_j}; for two, it's the sigmoid. You might expect the gradient of "log of a ratio of exponentials, fed through a log-loss" to be a horror. It isn't. When you compose softmax with cross entropy and differentiate with respect to the logit z_i, an almost magical cancellation happens and the whole thing collapses to

\frac{\partial L}{\partial z_i} = q_i - p_i,

the predicted probability minus the target. Prediction minus truth — the error itself, nothing more. No awkward factors, no vanishing derivatives when the model is badly wrong; the gradient is largest exactly when the mistake is largest. The same clean form is what makes the sigmoid-plus-log-loss pairing in logistic regression so well-behaved. This is no coincidence: cross entropy and softmax are a matched pair, engineered (or rather, discovered) so that the loss and the squashing function's nonlinearities annihilate each other and leave behind the simplest possible learning signal.

Feel it move

Formulas are one thing; watching the loss lurch is another. The cross-entropy playground has two panels. In the first, drag the predicted probability of the true class and watch the -\log p curve — see how gently the loss falls once you're confident and correct, and how violently it spikes as you drift toward a confident mistake. In the second, drag the bars of a predicted distribution against a fixed one-hot truth and watch cross entropy and KL divergence update together, hitting bottom exactly when your bars line up with reality. It's the fastest way to feel why "just match the true distribution" and "minimize cross entropy" are the same instruction.