A key wrapped in a key wrapped in a key

The two parity essays in this little series are about keeping bits intact — surviving disks that die. This one is about keeping them secret — surviving disks that walk. And it opens on a small mystery: you set up an encrypted ZFS dataset by typing a single passphrase, yet the moment you look under the hood there are three different keys in play, arranged in a chain where each one guards the next. It looks like over-engineering. It is the opposite: every link in that chain buys back a specific operational headache that flat, one-key encryption cannot. This essay follows the chain link by link, then goes under it to the field arithmetic that authenticates every block, and ends where honest crypto essays should — with what the scheme deliberately does not protect.

If you've read the LUKS essay, the opening beat will feel familiar: LUKS also refuses to let your passphrase encrypt the disk directly, for the same reason. ZFS takes the idea one level deeper.

Three keys, not one

Here is the whole hierarchy, top to bottom:

1. The wrapping key (a KEK). This is the only key derived from something you provide. Give ZFS a passphrase and it runs it through PBKDF2 — a deliberately slow, salted hash — to stretch it into a 256-bit key-encryption key, or KEK. (You can instead hand ZFS a raw 32-byte key, or point it at a key file or a command that fetches one; keyformat and keylocation decide.) The defining feature of the KEK is what it doesn't do: it never touches your data. Its entire job is to encrypt one other key.

2. The master key (wrapped). When you create the encrypted dataset, ZFS generates a random master key from the kernel's RNG — uniform, never derived from your passphrase, never stored in the clear. The KEK wraps it: encrypts it with an authenticated key-wrap (AES-256, in the RFC 3394 lineage) and stores the resulting blob in the dataset's metadata. To "load the key" for a dataset is exactly this: derive the KEK from your passphrase, unwrap the blob, check the built-in integrity tag, and hold the master key in kernel memory. A wrong passphrase produces a wrong KEK, the unwrap's integrity check fails, and ZFS rejects it — it never has to risk decrypting real data to tell you that you mistyped.

3. The data keys (DEKs). The master key still doesn't directly encrypt blocks. Instead ZFS derives per-purpose data-encryption keys from it with HKDF (an HMAC-based key-derivation function), and those keys drive the block cipher over your actual data. One master key fans out, deterministically, into the keys that do the bulk work.

Read bottom to top, the chain is: your passphrase → (PBKDF2) → KEK → (unwrap) → master key → (HKDF) → data keys → your blocks. Three keys, each guarding the next.

Why the indirection pays for itself

None of this is decoration. Each link removes a concrete pain that a single-key design would inflict:

Changing a passphrase is instant. Because your passphrase only guards the KEK, and the KEK only wraps the master key, a zfs change-key re-derives a new KEK and re-wraps the same master key — a few dozen bytes rewritten. The master key is unchanged, so not a single data block is re-encrypted. Rotate the human-facing secret on a 40-terabyte pool in the time it takes to hash a passphrase. A design where the passphrase-derived key encrypted data directly would have to rewrite the entire pool.

Inheritance has a clean root. Encryption is a dataset property that children inherit from an encryption root — the dataset where the key was actually loaded. Child datasets share the encryption root's key material through the same wrapping mechanism, so loading one key unlocks a whole subtree, and the boundary of "what one passphrase controls" is a precise, inspectable thing rather than an ad-hoc collection.

Encrypted backups to untrusted storage. This is the payoff that the whole structure exists to enable. zfs send --raw streams the dataset's blocks still encrypted, along with the wrapped master key — but not the KEK. The receiving side stores ciphertext it cannot read; it never sees your passphrase and never holds a key that could decrypt anything. You can replicate encrypted datasets to a cloud box, a friend's NAS, or a backup appliance you fundamentally distrust, and the confidentiality contract holds end to end. The remote can even scrub and resilver the data — parity and checksums operate on ciphertext just fine — without ever being able to read it. This is the feature that makes the three-key dance worth learning.

Under the hood: what encrypts a block

The data keys drive AES in an authenticated mode — AES-256-GCM by default (with CCM available). "Authenticated" is the important word and the real difference from full-disk encryption: recall that LUKS's XTS mode encrypts but does not authenticate — flip a ciphertext bit and it decrypts to silent garbage. ZFS refuses that bargain. Every encrypted block carries a message authentication code (MAC), and because ZFS stores checksums in the block pointer anyway, it has room to store that MAC where a plaintext checksum would otherwise live. Tamper with a block and the MAC check fails; ZFS treats it exactly like a failed checksum — which, thanks to the parity machinery, can then be reconstructed from a redundant copy. Confidentiality and integrity in the same pass.

GCM is worth unfolding one level, because its two halves are two ideas you've already met:

• Confidentiality is AES in counter (CTR) mode. AES encrypts a running counter to make a keystream, which is XORed with the plaintext. This is why the ciphertext is exactly the plaintext's size and why encryption parallelizes.

• Integrity is GHASH, a polynomial hash evaluated in — of all places — \mathrm{GF}(2^{128}), the same 128-bit finite field that carries LUKS's XTS tweak. The ciphertext blocks C_1, \ldots, C_m are read as coefficients of a polynomial and the MAC is essentially

  \mathrm{GHASH} = \sum_{i} C_i \, H^{\,m-i+1} \pmod{m(x)},

  evaluated at a secret point H = E_K(0) in the field, then combined with the encrypted counter. The RAIDZ2 essay built a finite field to make redundancy possible; GCM uses the very same kind of arithmetic to make forgery infeasible. It's striking how often "do algebra in a finite field" turns out to be the answer — parity, tweaks, and authentication tags all lean on it.

The nonce problem, and why COW dissolves it

GCM has one famous, unforgiving rule: never reuse an (key, IV) pair. Encrypt two different plaintexts under the same key and the same counter value and the keystreams are identical — XOR the two ciphertexts and the AES cancels, leaking the XOR of the plaintexts; worse, the authentication key H can be recovered, and forgery becomes possible. Nonce reuse is how GCM deployments die.

For an in-place block device this is a genuine hazard: rewrite sector n and you must somehow guarantee a fresh IV every single time, forever, or risk colliding with a past write to the same location. ZFS gets a structural free pass here, and it comes straight from the property that also closed the RAID write hole: ZFS is copy-on-write. It never overwrites a live block in place. A modified block is written to a new location with new metadata, and the old block is only freed once the new one is committed. Since IVs are derived per-block-write from block context rather than from a location that gets reused, "same key, same IV, different plaintext" essentially cannot arise in normal operation — the situation that would demand it, overwriting a block, is the one thing ZFS structurally never does. The architectural choice made for crash-consistency turns out to hand the cryptography exactly the guarantee it most needs. (There is a subtlety worth flagging honestly: this is why ZFS is careful with its IV/salt derivation and why key reuse across send/receive is handled explicitly — the free pass depends on the master key and IV derivation being managed correctly, not on luck.)

What it does not hide

Every honest encryption essay owes you the boundary of the promise, and ZFS's is specific.

It's confidentiality of block contents, not of structure. Native encryption protects file and directory data and names, and the values in blocks. It does not encrypt all pool-wide metadata: the existence and layout of datasets, their names, snapshot names, and properties, and the overall block-tree shape can remain visible to someone with raw access to the pool. An adversary who images your disks can often tell that you have a dataset called backups/tax-2019 and roughly how big it is, even if they can't read a byte inside it.

Sizes and timing leak. Block sizes, the number of blocks, and how they change between snapshots are observable, so activity patterns and approximate file sizes can leak even under perfect encryption of contents.

Dedup interacts with encryption carefully. Deduplication works by spotting identical blocks — which requires comparing something derived from their contents. Encrypted dedup is possible in ZFS but constrained (it must key the comparison in a way that doesn't leak across datasets), and it necessarily reveals within its scope that two blocks are identical. If your threat model cares about that, it's a real consideration, not a footnote.

A --raw backup is only as revocable as your key discipline. Because the remote holds the wrapped master key alongside the ciphertext, anyone who later learns your passphrase (or KEK) can decrypt that backup. Rotating your passphrase re-wraps your copy's master key, but a raw stream already sent carries its own wrapped master key — revocation of an exfiltrated backup means the master key itself must be considered compromised, which is a much bigger operation than a passphrase change.

Both halves are algebra

Stand back and the series closes on a single observation. RAIDZ1 kept bits intact with one equation over the field of bits; RAIDZ2 survived two failures by moving to \mathrm{GF}(2^8); ZFS encryption keeps bits secret with a key hierarchy whose bulk cipher authenticates every block using a polynomial hash over \mathrm{GF}(2^{128}). Redundancy and secrecy look like different departments, but underneath they run on the same machinery: finite fields, one careful equation at a time. ZFS's real trick is that it made both guarantees fall out of the same copy-on-write, checksum-everything design — the parity locates the failure, the checksum-shaped MAC catches the tampering, and the passphrase you typed at the top is three keys away from the bytes it protects.