What Is a Margin of Error? The Number That Is Almost Always Understated
By the BrainSnail editorial team. How these articles are written and checked, and how to tell us when one is wrong.
A poll reported with a margin of three points sounds precise, and the figure describes only one source of uncertainty, the randomness of drawing a sample. Every other thing that can go wrong in a survey lies outside it, and those other things are usually larger.
What the figure covers
The margin of error quantifies sampling variability, meaning how much a result would differ if the same survey were repeated with a different random sample drawn from the same population. It is calculated from the sample size and the observed proportion, conventionally at a ninety-five percent confidence level, which means that if the procedure were repeated many times, the interval would contain the true value in about ninety-five percent of them. For a sample of a thousand and a proportion near half, it comes out at roughly three percentage points. Two consequences follow immediately and are routinely ignored in reporting. The figure applies to the whole sample, so any subgroup, meaning young voters or a particular region, has a much larger margin because it rests on far fewer responses. And it applies to a single proportion, so the difference between two candidates has an uncertainty larger than the margin quoted, roughly twice as large when comparing shares within the same poll.
What it does not cover
Every other source of survey error sits outside the calculation and cannot be quantified by it:
- •Coverage error, where the method cannot reach part of the population at all, which was the flaw in famous historic polling failures
- •Non-response bias, where those who answer differ systematically from those who do not, which has grown into the dominant problem as response rates have fallen to a few percent in many countries
- •Question wording and order, which measurably shift answers and are chosen by the pollster
- •Interviewer effects and mode effects, since the same person answers differently to a human, a recorded voice and a web form
- •Weighting and modelling decisions, particularly assumptions about who will actually turn out to vote, which can move a headline figure by several points and differ between firms using identical raw data
- •Late changes of mind, which no survey conducted before an event can capture at all
How much bigger the real uncertainty is
Researchers have compared final pre-election polls with results across many elections and found that the actual error is roughly twice what the quoted margins imply, which means a poll advertising three points behaves in practice more like six. That analysis, based on large historical datasets, supports a simple rule for readers: treat the stated margin as a lower bound rather than a total. It also explains a pattern that otherwise looks like incompetence, since polls that miss by five or six points are within the realistic error even though they sit outside the advertised one. Polling averages help by cancelling some independent errors between firms, and they do not help against errors that all firms share, which is exactly what happens when the entire industry uses similar assumptions about who will vote. That shared component is why polling misses tend to be correlated across an industry rather than randomly distributed, and why an average can be confidently wrong.
Reading a poll properly
A small number of habits substantially improve interpretation. Check the sample size and whether the reported figure concerns a subgroup, since a headline about a demographic in a national poll may rest on two hundred people. Treat a difference smaller than the margin as no measured difference rather than as a narrow lead. Look at the trend across many polls rather than at any single one, since movement is more reliable than level. Note the mode and the dates, since a poll conducted over a weekend and one over a week reach different people. Read the question actually asked rather than the headline. Check who commissioned it, since advocacy polling exists and is legal. And expect the direction of error to be unknown, since knowing that polls are typically off by several points says nothing about which way, which is why confident predictions from narrow leads are unjustified regardless of how many polls agree.
The takeaway
The quoted figure measures only the randomness of drawing a sample, and it applies to the whole sample and to a single proportion, so subgroups and gaps between candidates carry larger uncertainty. Non-response, weighting choices and turnout assumptions sit entirely outside it. Historical comparison finds real error running about twice the advertised margin, and errors are correlated across firms because they share assumptions.