What Is Survivorship Bias? Studying Only the Things Still Here

During the Second World War, Allied analysts examined returning bombers to decide where to add armour, and the damage was concentrated on wings, fuselage and tail. The apparent conclusion was to armour those areas. Abraham Wald, working for the Statistical Research Group, pointed out that the sample consisted only of aircraft that had come back, so the distribution of damage showed where a plane could be hit and still return. The armour belonged where the returning planes showed no damage, principally the engines and cockpit, because aircraft hit there did not come back to be examined. The story is frequently told loosely and the underlying work is real, consisting of a rigorous analysis of how to estimate vulnerability from incomplete data. What makes it the standard illustration is that the missing observations were not merely absent but were precisely the informative ones, and that the intuitive reading of the data pointed exactly the wrong way.

The pattern appears wherever a selection process runs before observation, which is almost everywhere:

• •Business advice drawn from successful companies and founders, since the same strategies were followed by many firms that failed and are not available to be interviewed, which makes most success literature uninterpretable
• •Investment fund performance, since funds that perform badly are closed or merged and disappear from the index, inflating the apparent average return of the ones remaining
• •The belief that old buildings, furniture or music were better made, since the poor examples were demolished, discarded or forgotten and only the best survived to be admired
• •Medical treatments judged by the patients who returned for follow-up, who differ systematically from those who did not
• •Historical evidence generally, since documents survive selectively and the impression of a period is formed from what lasted
• •Advice from people who took a risk and succeeded, including dropping out of education or ignoring conventional guidance, which systematically omits everyone who did the same and did not

The bias is unusually durable because it does not feel like missing information. A dataset of survivors looks complete: the rows are there, the numbers are real, and the analysis can be done carefully and still be wrong. It also interacts with narrative, since the human preference for explanatory stories is satisfied by an account of why the survivors succeeded, and a story requires characters who are present. Hindsight makes the outcome feel inevitable and therefore attributable to the visible characteristics. And the incentives run one way, because successful people are available to write books and give talks while unsuccessful ones are not, so the supply of survivor-based evidence is enormous and the supply of the counterfactual is close to zero. Related errors compound it, including the base rate neglect that ignores how many attempts produced the observed successes, and publication bias in research, which is survivorship bias applied to studies rather than to people.

The practical defence is a habit of asking what has been filtered out before the data reached you, and then trying to find it. Concretely that means asking what the denominator is, since a claim about successful people is meaningless without knowing how many attempted the same thing; deliberately seeking the failures, which frequently requires different sources such as bankruptcy records, discontinued product lists or fund closure data; checking whether a dataset was assembled by a process that removes cases, including attrition in a long study or funds vanishing from an index; and treating any conclusion drawn from a group defined by its outcome with particular suspicion, since selecting on the outcome is the sharpest form of the problem. In research, prospective designs that follow everyone from the start avoid it by construction, which is one of the main reasons they are preferred to retrospective ones despite costing far more.

Survivorship bias arises when a group has been filtered before observation, so the survivors describe the filter rather than the population, and the missing cases leave no trace in the data. Wald's analysis of returning bombers showed the armour belonged where survivors were undamaged. It corrupts business advice, fund performance figures, judgements about the quality of the past and any conclusion drawn from people selected by their outcome. The defence is asking what the denominator was and deliberately going to find the failures.