What Is Statistics? Making Sense of Data When No Single Measurement Can Be Trusted

The first job is to summarise. A set of numbers has a centre and a spread, and both need stating. The mean, the total divided by the count, is the familiar average and is pulled about by extremes: the mean income of a room is doubled when a billionaire walks in. The median, the middle value when the numbers are sorted, is not, which is why house prices and incomes are reported as medians. The spread is described by the standard deviation, roughly the typical distance of a value from the mean, and by the range between the quartiles. Much of the data in nature and in life falls into the bell-shaped normal distribution, in which about two thirds of values lie within one standard deviation of the mean and 95 percent within two, and a great deal of statistical method rests on that shape, which is why the cases that do not follow it, incomes, city sizes, earthquake magnitudes and stock market crashes, cause so much trouble.

The second job is to reason from a sample to the whole. A poll of a thousand voters can estimate the vote of forty million to within about three percentage points, not because a thousand is a large fraction of forty million but because the error of a random sample depends only on the sample's size, shrinking as the square root of it. That is the central fact of the field, and the word random carries the weight: a sample of a million chosen badly, as the Literary Digest's postal poll of car and telephone owners was in 1936, is worse than a thousand chosen well, and the Digest predicted a landslide for the candidate who lost. The confidence interval that accompanies an estimate states the range within which the truth probably lies, and a result without one is a guess with a decimal point.

The third job is to decide whether an effect is real. A drug group recovered a little faster than a placebo group; could that be chance? The standard test asks how often a difference at least that large would arise if the drug did nothing, and reports the answer as a p-value; by convention a p below 0.05, one in twenty, is called significant. The convention has caused more confusion than any other in science, and the things a p-value is not:

• •It is not the probability that the drug does nothing; it is the probability of the data if it does nothing, which is a different thing
• •It is not a measure of how large or important the effect is; a trivial effect in a huge sample can have a tiny p
• •It is not a guarantee: one in twenty true-nothing experiments will pass, and a researcher who runs twenty analyses and reports the one that passed has found nothing
• •It is not a threshold nature respects; 0.049 and 0.051 are the same evidence

Most statistical deception is not lying but choosing. A graph whose vertical axis starts at 90 rather than zero turns a two percent change into a cliff; a relative risk, doubled from one in a million to two in a million, sounds far graver than the absolute risk it describes; a correlation between ice cream sales and drowning is real and is caused by summer, not by ice cream; and a hospital with the highest death rate may simply take the sickest patients. Survivorship bias, judging by the cases that made it through, is why the aircraft that returned from raids in 1943 were armoured where they had not been hit, on the statistician Abraham Wald's advice, since the ones hit there had not returned. Simpson's paradox, in which a trend in every group reverses when the groups are combined, once made Berkeley's admissions look biased against women overall while favouring them in most departments. Learning statistics is largely learning to ask what was left out.

In 2015 a group of psychologists repeated a hundred published experiments and found that only about a third gave the same result, and similar exercises in medicine, economics and cancer biology have found the same. The causes are statistical: small samples, which produce large and unreliable effects; the freedom to analyse data many ways and report the one that worked; and journals that publish positive findings and not the negative ones that would balance them. The remedies are also statistical: registering the analysis before the data are collected, larger samples, reporting effect sizes and intervals rather than a bare p, and the older methods of the Reverend Bayes, revived by computers, which update a prior belief by the evidence and give the probability people actually wanted. A discipline invented in the seventeenth century to price annuities and settle gambling arguments is now the referee for what counts as knowledge.

Statistics is the mathematics of drawing conclusions from data that vary: describing it with means, medians and spreads; inferring from random samples, whose accuracy depends on size rather than on the population; and testing whether effects are real with p-values that state the chance of the data under no effect and nothing more. Its misuse, through chosen axes, relative risks, confounded correlations and selective reporting, is why a third of published findings do not replicate, and its proper use is the only defence.