Why Are a Few Words So Common? A Pattern That Holds Everywhere

Counting how often each distinct word appears in a large text produces an extremely uneven distribution. The commonest word typically accounts for around six or seven per cent of all words, the top ten cover roughly a quarter, and the top hundred cover close to half, which means a reader who knows a few hundred words recognises most of the words on a page and understands very little, since the common ones are grammatical rather than substantive. At the other end, a large proportion of the distinct words in any text appear exactly once, and that proportion stays high no matter how much text is added, because new rare words keep arriving. The pattern is the same in every language examined.

The distribution follows a specific mathematical form closely:

• •Ranking words by frequency, the frequency is roughly inversely proportional to the rank
• •The second commonest word appears about half as often as the first, the third a third as often, and so on
• •Plotting frequency against rank on logarithmic axes gives approximately a straight line
• •The relationship is named after George Zipf, who popularised it in the 1930s and 1940s
• •It fits the middle of the range well and deviates at both extremes
• •The same form appears in city sizes, website visits, firm sizes and much else

Explanations have been proposed for eighty years and none commands agreement. Zipf proposed a principle of least effort, in which speakers economise by reusing words and hearers need enough distinct words to be understood, with the distribution being the compromise. A mathematical objection is that random text generated by typing letters and spaces at random produces a similar distribution, which was demonstrated by Benoit Mandelbrot and suggests the pattern may carry no information about language at all, though the details of the fit differ. Preferential attachment accounts, where a word already common is more likely to be used again, generate the form directly. The honest position is that several mechanisms produce the pattern and identifying which operates in language has proved difficult.

The distribution splits vocabulary into two populations that behave quite differently. Function words, meaning articles, prepositions, pronouns and auxiliaries, are few in number, enormously frequent, closed to new members and carry grammatical rather than descriptive meaning. Content words, meaning nouns, verbs, adjectives and adverbs, are numerous, individually rare, open to new members constantly and carry what a text is about. Removing function words from a text leaves it largely comprehensible, which is why headlines and telegrams drop them and why early search engines ignored them entirely. Removing content words leaves nothing. That division also explains why the most frequent words are useless for identifying a topic and are excellent for identifying an author, since nobody chooses them deliberately.

The distribution matters practically well beyond linguistics. Language teaching uses frequency lists to order vocabulary, on the reasoning that learning the commonest few thousand words covers the great majority of ordinary text, which is measurably true and is why such lists are the backbone of course design. Search and compression algorithms exploit it, since assigning short codes to frequent items and long ones to rare items is exactly what compression does. Authorship attribution uses the frequencies of common function words, which authors use at stable individual rates without noticing, and the technique has been applied to disputed texts with some success. And the long tail of words appearing once is why language models require enormous training data.

The commonest word takes six or seven per cent of a text and the top hundred cover close to half, while a large share of distinct words appear exactly once however much text is added. Frequency is roughly inversely proportional to rank, which is regular enough to be a law. Random text produces a similar pattern, which is why what causes it in language is still disputed.