Why Did the Bug Vanish When You Looked for It? Two Things Happened at Once

Two parts of a program running at the same time both read a shared value, both calculate a new value from what they read, and both write their result back. Each one behaves correctly in isolation. Together they can produce an answer that is simply wrong, because the second wrote its result based on a value that was already stale, and one of the two updates vanishes. Nothing crashed and no rule was broken. The outcome depends entirely on the relative timing of the two, which is decided by the operating system and is not under the program's control.

Almost every property of the fault works against the person hunting it:

• •It appears only for particular interleavings, which are rare
• •It cannot be reproduced reliably, so testing frequently passes
• •Adding logging changes the timing and may hide it
• •Running under a debugger slows things and may hide it
• •It appears under load, which means in production and not in testing
• •The symptom usually surfaces far from the code that caused it

The solutions all amount to removing either the sharing or the simultaneity. A lock makes one part wait while another works on the shared value, which is correct and costs performance, and introduces its own failure where two parts each hold what the other needs and neither can proceed. Certain operations can be made indivisible by the hardware, so a read, a calculation and a write happen as a single uninterruptible step. Better still, the shared value can be eliminated, by giving each part its own copy or by passing messages instead of sharing memory, which is why some languages are designed to make sharing difficult.

The same structure appears outside programming wherever two independent processes touch the same thing. Two people editing the same document and saving in sequence lose one set of edits, which is why collaborative editors were built to merge rather than overwrite. Two cash machines checking the same balance before dispensing can both approve. A booking system can sell the last seat twice. Even a shell script that checks whether a file exists and then creates it can be beaten by another copy of itself. Recognising the shape is more useful than memorising any particular fix.

The class of fault has a serious safety record and the best documented case is instructive. A radiation therapy machine in the mid 1980s killed several patients because a fast typing operator could reach a particular sequence before a slow subroutine had finished updating a setting, leaving the machine in a state its designers believed impossible. A large power failure in North America in 2003 spread further than it should have because a monitoring system stalled on a fault of this kind. Security flaws frequently rest on the same idea, where a program checks whether a file is safe and then opens it, and something changes the file in between.

When two simultaneous operations read and write the same value, the result depends on timing the program does not control, so one update can silently vanish. The fault is rare, unreproducible, and hidden by logging and debuggers, which is why it survives testing and appears under load. Fixes remove either the sharing or the simultaneity, and the failure has killed people.