Logistic regression: bending a line until it becomes a probability

Suppose the question isn't "how much?" but "which one?" Will this email turn out to be spam or not; will the tumour be benign or malignant; will the customer click or scroll on by. The answer is a label, a coin that lands one way or the other, and your job is to guess the bias of that coin from the features you can see. It sounds like a small change from linear regression — swap a number for a yes-or-no — but that small change breaks the machinery, and the repair is one of the prettiest little constructions in statistics.

The name is a historical booby-trap, by the way. Logistic regression is a classification method; it earns the word "regression" only because, under the hood, it's still fitting a linear function to your data. The linear part never leaves. We just teach it some manners.

Why a straight line won't do

Try the naive thing first, because seeing it fail is instructive. Encode the two classes as y \in \{0, 1\} and fit an ordinary least-squares line \hat y = x^\top\beta, hoping to read the output as "the probability that y = 1." The trouble is immediate: a line doesn't stop at 1, and it doesn't bottom out at 0. Push x far enough to the right and the line predicts a probability of 1.4; push left and it promises you -0.3. Those aren't probabilities, they're nonsense — a coin can't land heads 140% of the time.

You could clip the output to [0,1] by brute force, but that just papers over the real problem: the shape is wrong. A line assumes every extra unit of x buys you the same extra chance of y=1, forever. Real yes/no problems don't behave like that. Once you're already 99% sure something is spam, more evidence should barely move the needle — you're pressed against the ceiling. What we want is a response that is nearly flat near the extremes and steep in the uncertain middle. We want an S, not a slash.

The log-odds, and the squash that follows

Here's the move. Instead of forcing the line to be the probability, we let the line be something that lives naturally on the whole number line, and then map that thing into [0,1].

The something is the log-odds. The odds of an event with probability p are \frac{p}{1-p} — the ratio a bookmaker quotes, running from 0 (impossible) up through 1 (an even bet) to +\infty (a sure thing). Take the logarithm and you get a quantity, the logit, that ranges over all the reals, from -\infty to +\infty. That is exactly the home a linear function wants to live in. So we declare:

\log \frac{p}{1-p} = x^\top\beta.

That's the whole model, stated as a claim about log-odds being linear. Now just solve it for p to see what it says about probabilities directly. Exponentiate both sides, giving \frac{p}{1-p} = e^{x^\top\beta}, write z = x^\top\beta for brevity, and unravel:

p = (1-p)\,e^{z} \;\Longrightarrow\; p\big(1 + e^{z}\big) = e^{z} \;\Longrightarrow\; p = \frac{e^{z}}{1 + e^{z}} = \frac{1}{1 + e^{-z}}.

That last expression is the logistic sigmoid, written \sigma:

p = \sigma(x^\top\beta) = \frac{1}{1 + e^{-x^\top\beta}}.

Read the picture and the two directions of the same idea click together. The forward map \sigma takes the unbounded linear score and gently squashes it into a valid probability — nearly 0 for very negative scores, nearly 1 for very positive ones, and passing through 0.5 at z=0. The inverse map, the logit, takes probabilities back out to the full number line. Logistic regression is linear in the log-odds; the curviness you see in the probabilities is entirely the sigmoid's doing. We kept the honest linear model and simply chose the right coordinate to be linear in.

The loss: from Bernoulli to cross-entropy

We have a model with a knob, \beta. Now we need a way to score how well any setting of that knob explains the data, so we can chase the best setting. As with least squares, the principled route is maximum likelihood — ask which \beta makes the data we actually observed as probable as possible.

Each label is a single flip of a biased coin, a Bernoulli trial. If our model says the probability of y_i = 1 is p_i = \sigma(x_i^\top\beta), then the probability it assigns to the observed label y_i can be written in one slick line:

P(y_i \mid x_i) = p_i^{\,y_i}\,(1 - p_i)^{\,1 - y_i}.

Check the trick: when y_i = 1 the second factor becomes (1-p_i)^0 = 1 and you're left with p_i; when y_i = 0 the first factor collapses to 1 and you're left with 1 - p_i. One formula, both cases. Assuming the observations are independent, the likelihood of the whole dataset is the product over all n points, and — exactly as in least squares — we take the logarithm to turn that unwieldy product into a friendly sum. Maximizing the log-likelihood is the same as minimizing its negative, so define the loss as the negative log-likelihood:

L(\beta) = -\sum_{i=1}^n \Big[\, y_i \log p_i + (1 - y_i)\log(1 - p_i) \,\Big], \qquad p_i = \sigma(x_i^\top\beta).

This is the cross-entropy loss, and it is worth pausing on how sensible it is. If the true label is 1 and you confidently predicted p_i near 1, then \log p_i \approx 0 and you pay almost nothing. But predict p_i near 0 when the truth is 1 and -\log p_i rockets toward +\infty — confident and wrong is punished mercilessly. Where squared error grows only quadratically, cross-entropy's penalty for a certain, incorrect claim is unbounded. It is a loss with a long memory for arrogance. (It has an information-theoretic backstory too, which gets its own essay: cross-entropy.)

No closed form — so we roll downhill

Here's where the story parts ways with its ancestor. Least squares handed us the normal equations, a single matrix solve that lands on the exact minimum in one shot. Set the gradient to zero and read off \hat\beta. Try the same thing here and you hit a wall: the cross-entropy gradient, set to zero, gives a system that is transcendental — the sigmoid tangles \beta up inside an exponential, and there is no rearranging it into a clean \hat\beta = (\text{stuff}). No formula exists.

The good news is that all is not lost, because the loss is convex. Cross-entropy composed with the sigmoid curves upward everywhere in \beta — one bowl, one bottom, no deceptive local dips to get stranded in. We can't jump to the minimum, but we can always tell which way is downhill, so we take the other road from the least-squares essay: roll to the bottom by gradient descent. And the reward for setting the problem up this carefully is a gradient of startling cleanliness. Stack the predictions into \sigma(X\beta) and the whole thing is:

\nabla_\beta L = X^\top\big(\sigma(X\beta) - y\big).

Look at what survived. It is X^\top times (prediction minus truth) — the exact same shape as the least-squares gradient X^\top(X\beta - y), with the bare linear prediction X\beta simply replaced by the squashed one \sigma(X\beta). All the exponentials and logarithms from the sigmoid and the log-likelihood conspire, in the derivative, to cancel almost perfectly and leave a plain residual. That is not a coincidence — it's a property of the whole family these losses belong to — but it never stops feeling like a small gift. Feed that gradient to the optimizer, step against it until the residual stops shrinking, and you have your \hat\beta.

The boundary is a straight line after all

For all the sigmoid's curviness, the decision it makes is bracingly simple. You classify a point as class 1 when p = \sigma(x^\top\beta) \ge 0.5. But the sigmoid crosses 0.5 at precisely one place — where its input is zero. So the rule "predict 1 when p \ge 0.5" is exactly the rule "predict 1 when x^\top\beta \ge 0," and the frontier between the two verdicts is the set

x^\top\beta = 0,

which is a straight line (a hyperplane, in higher dimensions). The sigmoid governs how confident we are as we move away from that frontier — gently unsure near it, emphatically certain far from it — but the frontier itself is flat. Logistic regression is a linear classifier wearing a probabilistic coat.

Feel the S-curve steepen

The link between separation and confidence is the kind of thing that clicks the moment you can drag it. The logistic regression playground strings two classes along a single axis and fits the sigmoid live by gradient descent on exactly the cross-entropy above. Slide the classes apart and the curve steepens — the model grows emphatically sure far from the middle, the steepness coefficient climbs, and the loss falls toward zero. Let the classes overlap and the sigmoid flattens into a cautious ramp, hedging its bets while the cross-entropy readout rises: there is genuine doubt now, and the loss is honest about it. Throughout, the gold line marks the decision boundary — one flat cut where the S-curve crosses 0.5, no matter how curved everything around it looks.

Where this goes

Two classes was the warm-up. When the question has more than two answers — is this digit a 3, an 8, or a 5? — the sigmoid generalizes to the softmax, which turns a whole vector of scores into a set of probabilities that sum to one, with cross-entropy following it right along. That's the natural sequel: softmax. But the skeleton never changes from what we built here — a linear score, a squash into probabilities, and a likelihood you push uphill by rolling its negative down.