Hot Spares, Rebuilds and Why RAID Is Not Backup
Arrays do not usually die when a drive fails. They die in the hours afterward — during the rebuild, when redundancy is spent, every surviving drive is reading flat out, and the array is one hiccup from the end. Understanding that window changes how you configure spares, schedule scrubs, and think about what RAID is actually promising.
What a hot spare buys: time, subtracted
A hot spare is a powered, idle drive the controller claims the instant a member fails — rebuild starts at 3 a.m. without waiting for a human and a courier. It does not shorten the rebuild; it deletes the response gap in front of it, which for remote or weekend-adjacent hardware is routinely the larger number. The classic layouts follow: 8 bays as 7-drive RAID6 + spare, 12 bays as two 6-wide groups sharing one, distributed-spare designs on ZFS (draid) for the impatient. The discipline that keeps spares honest: replace the consumed spare promptly — an array running spare-less because nobody restocked is the same risk wearing a calmer dashboard.
Why rebuilds got scary: the size curve
Rebuild time scales with capacity; drive throughput did not keep pace. A 2TB drive rebuilt in hours; a 16TB member takes the better part of a day under load — sometimes multiples of that on busy parity arrays. During that window, single-parity math turns ugly: reading every bit of every survivor multiplies exposure to unrecoverable read errors, and at consumer URE ratings (1 per 10^14 bits ≈ 12.5TB), a large single-parity rebuild statistically expects one. Enterprise drives rate 10× better, which is half the reason they belong in arrays; double parity is the other half. This curve — not fashion — is why RAID5 retired from big-disk service and RAID6/RAIDZ2 became the floor.
Scrubs: rehearsing the rebuild
A scrub reads and verifies everything on schedule, surfacing latent sector errors while redundancy still exists to fix them. Skipped scrubs mean the rebuild becomes the first full read in months — and discovers the latent errors mid-crisis. Monthly scrubs (with patrol-read equivalents on hardware controllers) plus alerting on the results is cheap insurance measured against what it prevents.
The line RAID cannot cross
Every RAID level answers exactly one question: does the array survive drive hardware failures? Deletion, ransomware, filesystem bugs, controller misadventure, fire, theft and fat-fingered scripts all replicate instantly across perfect redundancy — the mirror faithfully mirrors the mistake. Backup answers the other question: can yesterday be retrieved? The two are not substitutes at any RAID level, any spare count, any vendor. Run arrays for continuity, backups for history, scrubs for honesty — and let each layer keep the promise it can actually make.