What Is a Null Result? The Finding That Nothing Happened and Why It Vanishes

A result showing no statistically significant effect is not evidence that no effect exists, and the distinction is the single most common error in reading such results. Failing to detect something can mean the effect is absent, and it can equally mean the study was too small to detect an effect that is present, that the measurement was too noisy, or that the design was poor. Distinguishing these requires knowing the statistical power of the study, which is the probability it would have detected an effect of a given size had one existed, and an underpowered study finding nothing tells you almost nothing at all. A well-powered study finding nothing is genuinely informative and can effectively rule out effects above a certain size. Methods exist for expressing evidence in favour of no effect rather than merely absence of evidence, including equivalence testing and Bayesian approaches, and their use is increasing.

Several pressures push these results out of the literature and they reinforce each other:

• •Journal preferences, since novel positive findings attract citations and negative ones historically did not
• •Author decisions, since researchers judge a null result unlikely to be accepted and often do not submit it, which accounts for much of the loss
• •Career incentives, since hiring and promotion reward publications and a study that yields nothing yields nothing to publish
• •Funding pressure, since a programme showing results is easier to renew
• •Narrative preference, since a paper with a clear finding is easier to write and to read
• •The flexibility of analysis, since a dataset can often be analysed in many ways and some path to significance may be found and reported

The consequences for the published record are serious and quantifiable. If only studies finding effects are published, the average published effect is larger than the true effect, sometimes substantially, and reviews that combine published studies inherit the inflation. Effects that do not exist can appear well supported by several published studies while the unpublished studies finding nothing sit in file drawers. Replication attempts then fail, which produced much of the replication crisis in psychology and medicine, where large systematic efforts found that a substantial proportion of prominent findings did not hold up. In medicine the stakes are direct, since suppressed trials showing no benefit or showing harm have led to treatments being adopted on an incomplete record, and several documented cases of that have driven regulatory change. Methods exist to detect the bias in a body of literature, including funnel plot asymmetry, and they indicate it is widespread.

Several experiments that found nothing are among the most consequential in the history of science, which is the strongest argument against treating such results as failures. The attempt to detect the motion of the Earth through a supposed medium carrying light returned no effect, and that absence was one of the observations the theory of special relativity was built to explain. Searches for a fifth fundamental force, for proton decay and for various proposed particles have returned nothing and have progressively constrained the theories that predicted them, which is real progress even though no paper announced a discovery. Medical trials showing no benefit have removed widely used treatments from practice and prevented considerable harm. In each case the value came from the study being powerful enough that the absence meant something, which is exactly the condition that separates an informative null result from an inconclusive one.

The responses are structural rather than exhortative, which is the right approach given that the incentives cause the problem. Preregistration requires the hypothesis and analysis plan to be recorded before data collection, which prevents flexible analysis after the fact and makes an unpublished null result conspicuous. Registered reports go further by having journals accept a study on the basis of its design before results exist, which removes the outcome from the publication decision entirely and has been adopted by a growing number of journals. Clinical trial registration is now mandatory in many jurisdictions and required by major journals, with results reporting obligations attached, and compliance is monitored publicly. Journals dedicated to negative results exist with limited uptake. Preprint servers and open data lower the barrier to making results available regardless of a journal decision. The direction of travel is clear and the pace varies enormously between fields.

Finding no significant effect can mean no effect exists or that the study was too small to detect one, and without knowing the statistical power the two are indistinguishable. Authors mostly do not submit such results, which is where most of the loss occurs. The published record then overstates effect sizes and supports findings that fail replication. Registered reports remove the result from the publication decision.