The File Opened Fine Last Year. Nothing Touched It. Now It Is Corrupt

Information on any medium is held as a physical state that is not permanent. Magnetic domains on a disk lose alignment gradually and are disturbed by writes to neighbouring tracks. Charge held in flash memory leaks away through insulating layers, faster when warm and faster in cells that have been written many times. Optical discs suffer chemical degradation of the dye layer. Cosmic rays and background radiation flip individual bits in memory and occasionally on storage. None of these requires anybody to touch the file.

The dangerous property is that nothing announces the problem:

• •A changed bit is still a valid bit as far as the drive is concerned
• •Reading returns the wrong data with no error reported
• •Ordinary file systems do not verify what they return
• •So corruption is discovered only when somebody opens the file
• •Which may be years later, after every backup has copied it
• •Backups therefore preserve the corruption faithfully

Defending against silent corruption requires storing something that detects it, since detection is the hard part rather than repair. Checksums computed over each block and stored separately allow a system to know that what it read is not what was written. Storage arrangements that keep redundant copies or parity can then reconstruct the correct block from elsewhere. File systems designed for this verify every read against a checksum and repair automatically, and they also perform a periodic scan of everything stored, reading and verifying data nobody has asked for, precisely so that errors are found while a good copy still exists.

Estimating the rate is harder than it sounds and the published figures vary widely. Drive manufacturers quote an unrecoverable read error roughly once in every hundred million million bits, which sounds reassuring until the figure is compared against the size of a modern drive, at which point reading one entire drive gives a meaningful chance of hitting one. Large scale studies of storage systems have found error rates well above manufacturer figures, concentrated in a minority of drives rather than spread evenly. Memory errors from radiation are measurable in any large installation and are a standard reason for error correcting memory in servers.

The problem is worse for long term preservation than for everyday use, and institutions holding material for decades plan around it explicitly. The standard practice is to hold several copies in different places on different media, verify them against stored checksums on a schedule, replace any that fails from a good copy, and migrate everything to new media long before the old media reach their rated life. Format obsolescence compounds it, since a perfectly intact file in a format nothing can read is lost just as thoroughly, which is why archives also migrate formats deliberately.

Magnetic alignment, trapped charge and optical dyes all degrade without anybody touching the file, and the corrupted result reads back as valid data with no error reported, so backups copy it faithfully. Checksums stored separately are what make the problem detectable, and redundancy is what makes it repairable. Archives scan everything on a schedule rather than waiting for somebody to open a file.