Zipf’s Law for Word Frequencies: Word Forms versus Lemmas in Long Texts - Fig 3

<p>(a) Probability mass functions <i>f</i>(<i>n</i>) of the absolute frequencies <i>n</i> of words and lemmas in <i>La Regenta</i>, together with their fits, under rescaling of both axis. The collapse of the tails indicates the compatibility of both power-law exponents. (b) The same for, from top to bottom, <i>Artamène, Bragelonne</i> (both in French), <i>Seitsemän v., Kevät ja t</i>., and <i>Vanhempieni r</i>. (all three in Finnish). The rescaled distributions are multiplied in addition by factors 1, 10⁻², etc., for a clearer visualization.</p

Analysis of the association between random variables using Pearson and Spearman correlations as statistics.

<p><i>ρ</i> is the value of the correlation statistic and <i>p</i> is the <i>p</i>-value of a two-sided test with null hypothesis <i>ρ</i> = 0, calculated through permutations of one of the variables (the results can be different if <i>p</i> is calculated from a <i>t</i>–test). The sample size is</p

Coverage of the vocabulary by the dictionary in each language, both at the word-type and at the token level.

<p>The average for all texts is also included. Remember that we distinguish between a word <i>type</i> (corresponding to its orthographic form) and its <i>tokens</i> (actual occurrences in text).</p

<i>γ</i>_<i>l</i> (the exponent of the frequency distribution of lemmas) versus <i>γ</i>_<i>w</i> (the exponent of the frequency distribution of word forms).

<p>As a guide to the eye, the line <i>γ</i>_<i>l</i> = <i>γ</i>_<i>w</i> is also shown (solid line). Error bars indicate one standard deviation.</p

The fit of a linear model for the relationship between exponents (<i>γ</i>_<i>w</i> and <i>γ</i>_<i>l</i>) and the relationship between cut-offs (<i>a</i>_<i>w</i> and <i>a</i>_<i>l</i>).

<p><i>c</i>₁ and <i>c</i>₃ stand for slopes and <i>c</i>₂ and <i>c</i>₄ stand for intercepts. The error bars correspond to one standard deviation. A Student’s <i>t</i>-test is applied to investigate if the slopes are significantly different from one and if the intercepts are significantly different from zero. The resulting <i>p</i>-values indicate that in all cases the slopes are compatible with being equal to one. The intercepts are compatible with zero for the exponents, but seem to be incompatible for the cut-offs.</p

Size of vocabulary <i>V</i> (i.e., number of types) when texts are decomposed in different sorts of types, being these: word-lemma-tag (w-l-t), plain words, lemma-POS (l-pos), lemma-POS of words in the dictionary (l-pos dic), lemmas, and lemmas of words in the dictionary (lemma dic).

<p>The latter provide the most radical transformation, as it yields the largest reduction in resulting vocabulary.</p

Characteristics of the books analyzed.

<p>¹Clarissa: Or the History of a Young Lady.</p><p>²Moby-Dick; or, The Whale.</p><p>³El ingenioso hidalgo don Quijote de la Mancha (1605)—The Ingenious Gentleman Don Quixote of La Mancha (title in English); including second part: El ingenioso caballero don Quijote de la Mancha (1615).</p><p>⁴Artamène ou le Grand Cyrus—Artamène, or Cyrus the Great.</p><p>⁵Le Vicomte de Bragelonne ou Dix ans plus tard—The Vicomte of Bragelonne: Ten Years Later.</p><p>⁶Seven Brothers.</p><p>⁷Spring and the Untimely Return of Winter.</p><p>⁸The Story of my Parents.</p><p>⁹Madeleine and Georges de Scudéry.</p><p>The length of each book <i>L</i> is measured in millions of tokens.</p

The lower cut-off for the frequency distribution of lemmas (<i>a</i>_<i>l</i>) versus the lower cut-off for the frequency distribution of word forms (<i>a</i>_<i>w</i>).

<p>The line <i>a</i>_<i>l</i> = <i>a</i>_<i>w</i> is also shown (solid line).</p

Power-law fitting results for words and lemmas, denoted respectively by subindices <i>w</i> and <i>l</i>.

<p><i>V</i> is the number of types (vocabulary size), <i>n</i>_<i>m</i> is the maximum frequency of the distribution, <i>N</i>_<i>a</i> is the number of types in the power-law tail, i.e., with <i>n</i> ≥ <i>a</i>, <i>a</i> is the minimum value for which the power-law fit holds, and <i>γ</i> and <i>σ</i> are the power-law exponent and its standard deviation, respectively. 2<i>σ</i>_<i>d</i>, the double of the standard deviation <i>σ</i>_<i>d</i> is also given. <i>σ</i>_<i>d</i> is the standard deviation of <i>γ</i>_<i>l</i>−<i>γ</i>_<i>w</i> assuming independence, which is </p><p></p><p></p><p></p><p><mi>σ</mi><mi>d</mi></p><mo>=</mo><p></p><p></p><p><mi>σ</mi><mi>w</mi><mn>2</mn></p><mo>+</mo><p><mi>σ</mi><mi>l</mi><mn>2</mn></p><p></p><p></p><p></p><p></p><p></p>. The last column provides ℓ₁, the number of lemmas associated to only one word form. Notice that the lemma exponent is very close to the one found in Ref. [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0129031#pone.0129031.ref029" target="_blank">29</a>] for the tail of a double power-law fitting, except for <i>Moby-Dick</i> and <i>Ulysses</i>.<p></p

Probability density <i>D</i>(<i>n</i>_<i>l</i>/<i>n</i>_<i>w</i>) of the frequency ratio for lemmas and words, <i>n</i>_<i>l</i>/<i>n</i>_<i>w</i>, in <i>La Regenta</i>.

<p>Values of <i>n</i>_<i>l</i> smaller than <i>n</i>_<i>w</i> are disregarded, as they arise from words associated to more than one lemma. Bending for the largest <i>n</i>_<i>l</i>/<i>n</i>_<i>w</i> is expected as the maximum of the ratio is given by <i>n</i>_<i>l</i>, which is not constant for each distribution but has a variation of half an order of magnitude (see plot legend).</p