One extra disk, one equation

A disk in a server does not die politely. It does not send a letter. One morning the array is humming along with a petabyte of somebody's photographs on it, and one of the spinning platters simply stops answering — a head crash, a controller that browned out, a cable that a technician nudged with an elbow. The block you asked for is gone. And yet the array keeps serving files as if nothing happened, and a few hours later a replacement drive slides in and the missing data reappears, byte for byte, out of thin air.

It is not thin air. It is one equation, applied a few billion times. This essay takes RAIDZ1 — ZFS's single-parity layout, and morally the same trick as RAID5 — apart down to that equation, shows why the equation is exactly the right one and not merely a plausible one, and then hands you a live playground where you can shoot a disk and watch RAIDZ1 rebuild it. The sequel, RAIDZ2, asks what happens when two disks die at once, and the answer turns out to need a genuinely bigger idea — a finite field. But one death first.

The shape of the promise

Lay n disks side by side. RAIDZ1 stores your data across n-1 of them and reserves the last one for parity. A horizontal slice through all n disks — one chunk from each — is called a stripe, and the promise RAIDZ1 makes is per-stripe: lose any one disk in this stripe and I can reconstruct it from the other n-1.

The parity chunk is not a copy, and it is not a checksum. It is the bitwise XOR of all the data chunks in the stripe. Write D_0, D_1, \ldots, D_{n-2} for the data chunks — each one just a block of bits, say b bits wide — and the parity chunk is

P \;=\; D_0 \oplus D_1 \oplus \cdots \oplus D_{n-2},

where \oplus is XOR applied bit-by-bit down the whole width. That's the entire scheme. A single extra disk, holding the running XOR of the rest. The magic is that this one column lets you recover any of the columns, not just detect that something is wrong — and to see why, we have to be a little bit careful about what XOR actually is.

Why XOR, of all operations

XOR looks like a lucky accident until you notice it is the addition of a group, and then it looks inevitable. Take the set of b-bit blocks. Define "addition" as bitwise XOR. This structure (\{0,1\}^b, \oplus) satisfies every axiom you'd want:

It's closed and associative. XOR of two b-bit blocks is a b-bit block, and (a \oplus b) \oplus c = a \oplus (b \oplus c) because XOR is just independent addition mod 2 in each bit position, and addition mod 2 is associative.

It has an identity. The all-zeros block 0 leaves everything alone: a \oplus 0 = a.

Every element is its own inverse. This is the load-bearing fact. For any block a,

a \oplus a = 0,

because a bit XORed with itself is 0 in every position. There is no separate "subtraction" to remember: XOR undoes XOR. The group is abelian too — a \oplus b = b \oplus a — so the order in which you fold the data blocks together never matters.

That self-inverse property is the whole reconstruction argument, and it takes one line. Suppose disk k dies. You still hold every other data disk and the parity disk. XOR all of them together — the survivors and the parity:

\Big(\bigoplus_{i \neq k} D_i\Big) \oplus P.

Now substitute the definition of P and reorder freely (abelian, associative):

\Big(\bigoplus_{i \neq k} D_i\Big) \oplus \Big(\bigoplus_{i} D_i\Big) \;=\; \bigoplus_{i \neq k}\big(D_i \oplus D_i\big) \;\oplus\; D_k \;=\; 0 \oplus D_k \;=\; D_k.

Every surviving term appears twice — once from the survivors, once from inside P — and each such pair XORs to zero and vanishes. The only term that appears an odd number of times is D_k, the very disk you lost. It falls out alone. The same computation works if the disk that died was the parity disk itself: then you simply recompute P from the intact data. One equation, and it reconstructs any column you like. That is genuinely all RAIDZ1 does, billions of stripes over.

The fine print that changes everything

Here is the sentence people skip, and it is the most important sentence in the essay: the reconstruction above only works if you know which disk to reconstruct. Look again at the derivation. The term that survives is D_k — but k was given. We assumed we knew disk k was the dead one.

Coding theory has two words for two very different failures, and RAIDZ1 lives on the boundary between them:

• An erasure is a symbol that is missing at a known location. The drive reports an I/O error, or it's simply not there. You know exactly which chunk is gone; you just don't know its value.
• An error is a symbol that is silently wrong — a bit rotted on the platter, a firmware bug returned stale data, a cosmic ray flipped a cell — and nothing announced it. The location is unknown.

A single parity chunk can correct one erasure — that's the derivation above — but it cannot correct even one silent error. The reason is stark. If some disk hands back corrupted data D_j' = D_j \oplus e for an unknown error pattern e, then folding everything together gives you e itself, not a location. You learn that the stripe is inconsistent (the XOR of all n chunks, which should be 0, comes out nonzero), but you have no way to tell which of the n disks is the liar. One equation, one unknown you can pin down — and "which disk" is a second unknown. The algebra is out of room.

Classic RAID5 lives with exactly this gap, and it is why RAID5 has a bad name in circles that have been burned. RAID5 trusts the disks to report their own failures. When a disk lies quietly — returns wrong bytes with a cheerful "OK" — RAID5's parity can detect the inconsistency during a scrub but cannot locate it, and a naive rebuild will happily fold the corruption into the reconstructed disk, laundering a silent error into permanent data loss.

What ZFS adds: turning errors into erasures

This is the exact seam where ZFS stops being RAID5-with-nicer-tooling and becomes something with a different failure model. In ZFS, every block carries a checksum, and the checksum lives in the block's parent, not next to the data it guards. The pointer that tells ZFS where a block lives also records a (typically 256-bit) checksum of what should be there. So when a chunk is read back, ZFS doesn't trust the disk's opinion of its own health — it recomputes the checksum and compares.

The consequence is quietly profound: ZFS converts silent errors into erasures. A disk that returns rotted bytes fails its checksum, and now ZFS knows which chunk is bad — it has manufactured the "known location" that the parity equation demanded. At that point the single missing-and-known chunk is reconstructed by the one-line XOR above, the good value is written back over the rot (this is self-healing), and the read returns correct data to the application without it ever knowing there was a fight. The parity provides the value; the checksum provides the location; neither alone is enough, and that division of labor is the real design idea. It's the same instinct behind the checksum-guards-the-key move in LUKS — never trust the medium to tell you it's intact.

The write hole, and why COW closes it

There's one more classic RAID5 failure that's worth naming, because ZFS's answer to it is structural rather than bolted-on. Consider updating one data chunk in a stripe. The parity must change too, so a RAID5 write is really two writes — new data, new parity — and they cannot be truly simultaneous on separate spindles. If power dies between them, the stripe is left internally inconsistent: the parity no longer equals the XOR of the data. This is the write hole. The insidious part is that the inconsistency is invisible until another disk later dies and you go to rebuild from a parity that was already wrong — reconstructing garbage with total confidence.

Hardware RAID papers over this with battery-backed write caches. ZFS dissolves it. Because ZFS is copy-on-write, it never overwrites a live stripe in place. A modified stripe is written to fresh free space — data and parity together — and only once that whole new stripe is safely on disk does a single atomic pointer update swap it into the tree. There is no window in which a live stripe is half-updated: either the old stripe is current (and self-consistent) or the new one is (and self-consistent). RAIDZ also uses variable-width stripes for exactly this reason — a small write becomes a small, self-contained stripe rather than a read-modify-write of a full-width one. The write hole isn't patched; it's designed out of existence.

One is not always enough

RAIDZ1's arithmetic is beautiful precisely because it is so small: a group where every element is its own inverse, one parity column, one line of algebra to bring any disk back from the dead. Go kill a disk in the playground and watch the equation run — then try to kill a second one and watch it refuse.

That refusal is not a bug; it is a theorem. One equation determines one unknown. The moment two disks vanish at once — and in a large array rebuilding a failed disk stresses its siblings for hours, which is exactly when a second one tends to go — you have two unknowns and a single equation, and no amount of XOR will save you. To survive that, you need a second, independent equation. It turns out you cannot get one by XORing more cleverly; you have to leave the world of bits-mod-2 entirely and do arithmetic in a finite field. That's RAIDZ2, and it's where the real algebra starts.