Why Does a Small Number of Things Account for Most of the Total? Long Tails

Most people's intuition about variation comes from quantities like human height, which cluster around an average with symmetric variation and with extreme values so rare as to be practically impossible, so nobody is three times the average height. Many other quantities behave entirely differently, with most cases small, a few enormous, and no meaningful clustering around any typical value. The largest city in a country is not slightly larger than the second but several times larger. The richest person holds not a little more than average but a multiple of what most hold. In such distributions the average is dragged upwards by the extremes and describes almost nobody, so the median and the shape of the tail are what matter.

The same shape appears across domains with nothing else in common:

• •Wealth and income, which is where it was first described
• •City populations within a country
• •Word frequencies in any large body of text
• •Earthquake energies, where the relationship is a fundamental law of seismology
• •Sizes of files, firms and forest fires
• •Connections per node in many networks, including citations and web links

Several mechanisms generate this shape and identifying which applies in a given case is the interesting question. Proportional growth, where something grows by a percentage rather than a fixed amount and the percentage is random, produces it naturally over time. Preferential attachment, where new connections go to nodes that already have many, produces it in networks and explains why a few websites hold most of the links. Systems poised at a critical point produce it, which is the standard explanation for earthquakes and avalanches. Combining exponential processes produces it. The common thread is that the size already reached influences the growth to come, which is exactly what does not happen with height.

Recognising this shape in a set of numbers is a specific skill and doing it badly is common. Plotting the data on ordinary axes produces an uninformative spike against one edge, which tells nothing. Plotting the logarithm of the size against the logarithm of how many cases exceed that size turns the relationship into a straight line if the pattern holds, and the slope of that line characterises how heavy the tail is, which is the standard diagnostic. Fitting it properly requires care, since real data usually follows the pattern only above some threshold and the apparent straightness of a log plot is a weak test on its own. Published claims of this pattern have frequently not survived rigorous statistical testing, so the diagnostic is a starting point rather than a conclusion.

Treating such a quantity as though it clustered around an average produces systematic and expensive errors. Risk models that assume familiar bell-shaped variation understate the probability of extreme events by enormous factors, which is a documented contributor to financial crises, since a movement calculated as impossible under one assumption is merely uncommon under the other. Planning for average demand fails when demand is dominated by rare surges. Insurance pricing depends entirely on getting the tail right. Averages quoted for income, house prices or company size mislead unless accompanied by a median. And the informal claim that a fifth of causes produce four fifths of effects is a rough description of this shape rather than a rule, with the actual proportions varying enormously by case.

Many quantities have most cases small and a few enormous, with no clustering around a typical value, so the average is dragged upwards and describes almost nobody. Wealth, city sizes, word frequencies and earthquake energies all behave this way. Proportional growth, preferential attachment and critical systems all generate the shape, and treating such a quantity as bell-shaped understates extreme events by enormous factors.