What Is the Prosecutor's Fallacy? Confusing Two Very Different Probabilities

Suppose a forensic sample matches a defendant, and an expert testifies that the probability of such a match occurring by chance is one in a million. The prosecutor's fallacy concludes that there is therefore a one in a million chance the defendant is innocent. That does not follow, because the first figure is the probability of the evidence given innocence, and the second is the probability of innocence given the evidence, and converting between them requires knowing how many people could have been the source in the first place. If the relevant population is ten million, then roughly ten people would be expected to match by chance, so a match alone makes the defendant one of about eleven candidates rather than a near certainty. The same reasoning appears in medical testing, where a highly accurate test for a rare condition still produces mostly false positives, and it is the everyday face of Bayes's theorem, which specifies exactly how the prior probability must be combined with the evidence.

Several distinct errors travel together in courtrooms:

• •The defence fallacy, the mirror image, which argues that since ten people in the city would match, the evidence gives only a one in ten chance of guilt, ignoring all other evidence narrowing the pool
• •Multiplying dependent probabilities as if they were independent, which produces astronomically small and completely wrong figures when the characteristics are correlated
• •Using a database search match without adjusting for the fact that searching millions of profiles makes some match far more likely than testing one suspect
• •Treating an absence of evidence as evidence of absence without considering how likely the evidence would have been found if the event occurred
• •Base rate neglect generally, ignoring how common the thing being inferred is in the population before weighing the evidence
• •Presenting a likelihood ratio in words rather than numbers, where phrases such as strong support are interpreted very differently by different jurors

The most notorious example is the conviction of Sally Clark in England in 1999 for the murder of her two infant sons, both of whom had died suddenly. A paediatrician testified that the chance of two cot deaths in such a family was one in seventy-three million, a figure obtained by squaring the rate for a single death, which assumed the two events were independent when genetic and environmental factors make them plainly not. The figure was also presented in a way that invited the fallacy, since the relevant comparison was not between two cot deaths and nothing but between two cot deaths and two murders, which is also extremely rare. The Royal Statistical Society issued a public statement criticising the use of the statistic. Clark's conviction was quashed on appeal in 2003, partly on undisclosed medical evidence, and she died a few years later. Related prosecutions were reviewed. Comparable reasoning errors appeared in the case of Lucia de Berk in the Netherlands, a nurse convicted partly on a statistical argument about her presence at deaths, whose conviction was overturned after statisticians demonstrated fundamental errors in the calculation.

The reforms are partial. Forensic science has moved towards expressing results as likelihood ratios, stating how much more probable the evidence is under one hypothesis than another, which is the mathematically correct form and leaves the prior probability to the court where it belongs. Guidance for expert witnesses in several jurisdictions explicitly warns against stating the probability of guilt. English appellate courts have been notably cautious about statistical reasoning in general, in one judgment discouraging the use of Bayes's theorem for evidence that cannot be assigned reliable numbers, a position statisticians have criticised on the grounds that juries reason probabilistically anyway and do so worse without structure. Juror comprehension research finds that presentation format matters substantially, with natural frequencies, meaning statements about how many people out of a hundred, understood far better than probabilities expressed as percentages or odds. The underlying difficulty remains that the question a court must answer requires combining evidence with a prior, and courts are reluctant to make that prior explicit.

The probability of a match given innocence and the probability of innocence given a match are different numbers, and converting between them requires the size of the population who could have been the source. A one in a million match in a city of ten million leaves several candidates, not a near certainty. Sally Clark's conviction rested on a figure obtained by squaring a rate for events that are not independent, and it was quashed in 2003. Forensic evidence is now expressed as a likelihood ratio.