kaiyin.zhong

Writing

Essays

Things I have learned. Things I built, broke, and eventually figured out.

  1. Adam: the optimizer that keeps its own running notes Nov 14, 2017 · 8 min Plain gradient descent takes the same timid step in every direction and stalls the moment the landscape turns into a canyon. Adam fixes this by remembering — it keeps a running average of where the gradient has been pointing, and a separate running average of how big it's been, then uses both to size each step per parameter. Here's the ladder from SGD to momentum to RMSProp to Adam, why those early steps need a bias correction, and a live playground where four optimizers race across the same landscape at once. machine learning · math
  2. ReLU: the bent line that taught networks to go deep Sep 19, 2017 · 9 min The rectified linear unit is the most consequential function in modern deep learning, and it is embarrassingly simple: keep the positives, zero out the negatives. That one kink is doing enormous work. It's the nonlinearity that stops a deep network from collapsing into a single matrix, the gradient that refuses to vanish where sigmoids choke, and — summed a few hundred times — a machine that can bend a straight line into any shape you like. Here's why max(0, x) beat the smooth curves, how it dies, what the variants fix, and a live playground where you stack ReLUs into any curve you want. machine learning · math
  3. Softmax: turning opinions into odds Jul 18, 2017 · 7 min A neural network's last layer speaks in logits — raw, unbounded scores with no sense of shame. Softmax is the diplomat that takes those numbers and hands back a genuine probability distribution: positive, summing to one, and steeper wherever the scores disagree most. Here's where the formula comes from, why it doesn't care about a constant shift (and why that quiet fact is what keeps it from overflowing), how a single temperature knob slides it between argmax and a coin flip, why the two-class case is just the logistic sigmoid wearing a hat, and why pairing it with cross-entropy makes the gradient collapse to something almost embarrassingly clean. machine learning · math
  4. Cross entropy: the price of believing the wrong distribution May 30, 2017 · 7 min Cross entropy is the loss that trains almost every classifier you've ever met, and it isn't an arbitrary formula bolted onto neural nets — it falls straight out of information theory. Start with entropy as expected surprise, add KL divergence as the extra cost of using the wrong code, and cross entropy is just the sum of the two. Here's the derivation, why it's the same thing as negative log-likelihood, why it pairs so cleanly with softmax that the gradient collapses to (prediction − target), and a live playground where you drag a predicted distribution around and watch the loss react. machine learning · math
  5. Logistic regression: bending a line until it becomes a probability Apr 25, 2017 · 8 min You can't answer a yes-or-no question with a straight line — a line runs off to infinity in both directions, and probabilities are stuck between 0 and 1. Logistic regression fixes this with one elegant move: keep the linear part, then squash it through a sigmoid. Here's why the log-odds turn out linear, how the Bernoulli likelihood collapses into cross-entropy, why there's no tidy closed form this time, and how a single clean gradient carries the whole thing downhill. machine learning · math · statistics
  6. Gradient descent: the art of rolling downhill Mar 28, 2017 · 8 min Least squares hands you the answer in one clean matrix solve — but only because the bowl is a perfect quadratic and small enough to invert. Take away either luxury and you need a method that finds the bottom without ever seeing the whole landscape: take a step in the steepest downhill direction, then look around and do it again. That's gradient descent. Here's the update rule, why the learning rate is the one knob that makes or breaks it, when convexity guarantees you arrive, and a live playground where you crank η until the whole thing flies apart. machine learning · math
  7. Linear regression: the honest workhorse Feb 21, 2017 · 9 min Fit a line, and least squares hands you numbers. Turn it into a model — with a claim about the world, error bars on every coefficient, and a theorem that says you can't do better without giving something up — and you have linear regression. Here's the model it actually asserts, why its estimator is provably the best of its kind, what each coefficient really means, and the four quiet assumptions holding the whole thing up. machine learning · math · statistics
  8. Least squares: the line that argues with every point Jan 17, 2017 · 8 min Draw a line through a cloud of points and you've already made a choice about what 'best' means. Least squares makes that choice precise — and it turns out to be two stories at once: a calculus problem you can solve by setting a gradient to zero, and a geometry problem where the answer is a shadow. Here's both derivations, why we square the errors in the first place, and a live playground where you drag the points and watch the line fight back. machine learning · math · statistics
  9. The directions that matter Dec 20, 2016 · 10 min Given a cloud of data in high dimensions, which few directions carry almost all of the story? Principal component analysis answers it by finding the axes along which the data varies most — and those axes turn out to be exactly the top eigenvectors of the covariance matrix, or equivalently the top singular vectors of the data. It's the spectral theorem, positive-definiteness, and the SVD all cashing out at once. Here's the derivation, why the leftover directions are the ones safe to drop, and a playground where you reshape a point cloud and watch its principal axes swing to follow. linear algebra · machine learning · math · statistics
  10. Every matrix in three acts Nov 8, 2016 · 10 min There's one factorization that works for every matrix — square or not, invertible or not — and it says something almost unreasonable: whatever a matrix does, it does it as a rotation, then a set of independent stretches, then another rotation. That's the singular value decomposition. Here's the geometry that turns a circle into an ellipse, why the singular values are the square roots of eigenvalues in disguise, the Eckart–Young theorem behind every low-rank approximation, and a playground where you watch the three acts play out. linear algebra · machine learning · math
  11. Positive definite: the shape of a bowl Sep 13, 2016 · 9 min Feed a symmetric matrix a vector from both sides and you get a single number — and the sign of that number, for every vector at once, decides whether a surface curves up like a bowl, down like a dome, or twists into a saddle. Positive-definite matrices are the bowls, and they're everywhere machine learning lives: covariance matrices, Hessians, the loss surfaces you roll downhill. Here's the quadratic form, the tests for definiteness, and a playground where you bend a matrix from bowl to saddle and back. linear algebra · machine learning · math · statistics
  12. Eigenvectors and the spectral theorem Jul 5, 2016 · 10 min Most vectors get knocked off course when a matrix hits them. A precious few don't — they only stretch or shrink, never turn. Those are the eigenvectors, and they're the closest thing a matrix has to a set of natural axes. Here's what Av = λv really means, how diagonalization makes powers of a matrix trivial, the spectral theorem that hands symmetric matrices a perfect orthonormal set of axes, and a playground where you watch power iteration home in on the dominant direction. linear algebra · machine learning · math
  13. The shadow of a vector May 3, 2016 · 9 min Ask where a vector's shadow falls on a wall and you've asked the deepest question in applied linear algebra. Projection is how you find the closest point in a subspace to a point outside it — and the leftover, the part you couldn't reach, always lands exactly perpendicular. Here's projection onto a line and onto a subspace, the projection matrix, Gram–Schmidt and QR, and a playground where you drag a vector and watch its shadow and its perpendicular residual move with it. linear algebra · machine learning · math
  14. The four fundamental subspaces Mar 8, 2016 · 9 min Every matrix quietly carves its two worlds — input space and output space — into four subspaces, and once you see them, the whole question of 'does this system have a solution?' becomes a matter of geometry. Here's the column space and null space, the rank–nullity theorem that ties their sizes together, why solvability means 'b lives in the column space', and a playground where you pick a target and watch it fall inside or outside the reachable set. linear algebra · machine learning · math
  15. A matrix is a verb: what a linear map does Jan 19, 2016 · 9 min A matrix isn't a grid of numbers you memorize rules for — it's a verb. It takes a vector and moves it. Its columns tell you exactly where the basis vectors land, and that alone pins down everything the matrix does to everything else. Here's the linear map hiding behind the notation, why matrix multiplication is really function composition, what the determinant measures, and a live playground where you drag the columns and watch the whole plane bend with them. linear algebra · machine learning · math