Lexical shifts in the emergence of Zipf’s Law of Abbreviation in English

Zipf’s Law of Abbreviation is a famous statistical pattern whereby frequent words tend to be shorter than infrequent ones. Large-scale studies have found this pattern both in multi-billion word collections of texts from well-resourced languages (e.g. Piantadosi et al. 2010), as well as smaller collections of texts from close to a thousand languages (Bentz et al. 2016). And yet there are still glaring gaps in our knowledge of the fine details and origins of this pattern. This talk attempts fill these gaps through a careful investigation of the pattern in a historical corpus of English (COHA) supplemented by statistical and computational modelling.

The first case study looks at how the law of abbreviation seems to hold relatively stable over a period of 150 years in written texts at a large scale, while varying substantially across genres and parts of speech. A general pattern is that the law is stronger in genres that are closer to spoken language, which will be verified by testing two additional corpora of scripted and spontaneous spoken language. The second case study looks at how referring forms shift in response to changes in the frequency of the underlying concept (e.g. plane overtaking airplane as the frequency of the concept increases). This resource-intensive study uses LLMs to disambiguate words in context (e.g. plane ‘surface’ vs. plane ‘flying vehicle’) and a Monte Carlo resampling method to find non-trivial correlations between time series. A key finding is that the length of referring forms adapts to the frequency of the concept when that frequency increases, but does not show changes otherwise. Finally, I present simulation results that show how Zipf’s Law of Abbreviation may emerge in a simple model of communication even in the absence of any explicit pressure for short forms to be prioritised in frequent contexts.

Zoom: https://us06web.zoom.us/j/85250575838?pwd=M21okObuBY0UjWZgED4b18ms2O4JXr.1