Futrell, Richard

Hahn, Michael

English

ScholarWorks@UMass Amherst

Proceedings of the Society for Computation in Linguistics Volume 3 Article 53 2020 Crosslinguistic Word Orders Enable an Efficient Tradeoff of Memory and Surprisal Michael Hahn Stanford University, mhahn2@stanford.edu Richard Futrell University of California, Irvine, rfutrell@uci.edu Follow this and additional works at: https://scholarworks.umass.edu/scil  Part of the Computational Linguistics Commons Recommended Citation Hahn, Michael and Futrell, Richard (2020) "Crosslinguistic Word Orders Enable an Efficient Tradeoff of Memory and Surprisal," Proceedings of the Society for Computation in Linguistics: Vol. 3 , Article 53. DOI: https://doi.org/10.7275/zxj5-en69 Available at: https://scholarworks.umass.edu/scil/vol3/iss1/53 This Abstract is brought to you for free and open access by ScholarWorks@UMass Amherst. It has been accepted for inclusion in Proceedings of the Society for Computation in Linguistics by an authorized editor of ScholarWorks@UMass Amherst. For more information, please contact scholarworks@library.umass.edu. Crosslinguistic Word Orders Enable an Efficient Tradeoff of Memory andSurprisal (Abstract)Michael HahnStanford Universitymhahn2@stanford.eduRichard FutrellUniversity of California, Irvinerfutrell@uci.eduMemory limitations are well-established as afactor in human online sentence processing (Gib-son, 1998; Lewis and Vasishth, 2005), and havebeen argued to account for crosslinguistic wordorder regularities. For example, the Performance–Grammar Correspondence Hypothesis of Hawkins(1994) holds that forms which are easier to pro-duce and comprehend end up becoming part of thegrammars of languages. We build on expectation-based models of language processing (Levy, 2008)and on the theory of lossy compression (Coverand Thomas, 2006) to develop a highly generalinformation-theoretic notion of memory efficiencyin language processing, in terms of a trade-off ofsurprisal and memory usage. We derive a methodfor estimating a lower bound on the memory effi-ciency of languages from corpora, and apply ourmethod to corpora from 54 languages to test theidea that word order is structured to reduce pro-cessing effort under memory limitations. We findthat word orders tend to support efficient tradeoffsbetween memory and surprisal, suggesting thatword order rules are structured to enable efficientonline processing.Background Surprisal theory (Levy, 2008)posits that the processing effort on a word wtin context w1 . . . wt 1 is proportional to the sur-prisal of the word in context:S =   log P(wt|w1 . . . wt 1). (1)Experimental work has confirmed that surprisal isa reliable and linear predictor of processing ef-fort as reflected in reading times (Smith and Levy,2013).However, surprisal theory as presented abovecannot in principle account for effects of memorylimitations on online processing, because Equa-tion 1 represents surprisal as experienced by anidealized listener who accurately remembers theentire history of previous words w1...t 1. MoreFigure 1: Conceptual tradeoff between memory andsurprisal for two languages. In Language A (blue),a listener storing 1 bit can achieve average surprisal3.5, while the same level of surprisal requires 2 bitsof memory for a listener in Language B (red).realistically, human listeners deploy memory re-sources that maintain imperfect representations ofthe preceding context (Lewis and Vasishth, 2005;Futrell and Levy, 2017). Ifmt is a listener’s mem-ory state after hearing w1 . . . wt 1, then the truesurprisal experienced by the listener will be:SM :=   log2 P(wt|mt), (2)which must be larger than Eq. 1 on average (Coverand Thomas, 2006).Memory–surprisal tradeoff. These considera-tions imply a tradeoff between memory and sur-prisal: A listener maintaining higher-precisionmemory representations mt will, on average, in-cur lower surprisal, at the cost of higher memoryload. The idea of the memory-surprisal tradeoffis visualized in Fig. 1: for each desired level ofaverage surprisal, there is a minimum number ofbits of information which must be stored aboutcontext. The shape of the trade-off is determinedby the language, and in particular its word order:some languages enable more efficient trade-offsthan others by forcing a listener to store more bitsin memory to achieve the same level of average468Proceedings of the Society for Computation in Linguistics (SCiL) 2020, pages 468-469.New Orleans, Louisiana, January 2-5, 2020Afrikaans Amharic Arabic Armenian Bambara Basque Breton Bulgarian Buryat Cantonese Catalan Chinese Croatian Czech Danish Dutch English Erzya6.06.26.46.60.0 0.3 0.6 0.9MemoryUnigramCE − MedianEmpirical4.254.504.755.005.250.0 0.5 1.0 1.5 2.0MemoryUnigramCE − MedianEmpirical5.56.06.50 1 2 3MemoryUnigramCE − MedianEmpirical7.457.507.557.600.00 0.25 0.50 0.75 1.00MemoryUnigramCE − MedianEmpirical4.004.254.504.755.000.0 0.5 1.0 1.5 2.0MemoryUnigramCE − MedianEmpirical6.66.87.00.0 0.2 0.4 0.6MemoryUnigramCE − MedianEmpirical5.65.86.06.26.40.0 0.5 1.0 1.5MemoryUnigramCE − MedianEmpirical5.76.06.36.60.0 0.5 1.0MemoryUnigramCE − MedianEmpirical7.507.557.607.657.700.0 0.2 0.4 0.6MemoryUnigramCE − MedianEmpirical5.05.15.25.35.40.00 0.25 0.50 0.75 1.00 1.25MemoryUnigramCE − MedianEmpirical5.05.56.00 1 2MemoryUnigramCE − MedianEmpirical6.76.97.17.30.0 0.3 0.6 0.9 1.2MemoryUnigramCE − MedianEmpirical6.256.506.757.000.0 0.5 1.0MemoryUnigramCE − MedianEmpirical5.56.00.0 0.5 1.0 1.5 2.0MemoryUnigramCE − MedianEmpirical6.06.26.46.60.00 0.25 0.50 0.75MemoryUnigramCE − MedianEmpirical5.45.65.86.06.26.40.0 0.5 1.0 1.5MemoryUnigramCE − MedianEmpirical5.76.06.36.60.0 0.5 1.0 1.5MemoryUnigramCE − MedianEmpirical7.407.457.507.550.0 0.2 0.4 0.6 0.8MemoryUnigramCE − MedianEmpiricalEstonian Faroese Finnish French German Greek Hebrew Hindi Hungarian Indonesian Italian Japanese Kazakh Korean Kurmanji Latvian Maltese Naija6.56.66.76.86.90.0 0.2 0.4 0.6 0.8MemoryUnigramCE − MedianEmpirical5.86.06.26.40.0 0.5 1.0 1.5MemoryUnigramCE − MedianEmpirical5.65.86.06.20.0 0.3 0.6 0.9MemoryUnigramCE − MedianEmpirical4.55.05.56.00 1 2 3MemoryUnigramCE − MedianEmpirical5.76.06.30.0 0.5 1.0 1.5MemoryUnigramCE − MedianEmpirical6.506.757.007.250.0 0.3 0.6 0.9 1.2MemoryUnigramCE − MedianEmpirical6.006.256.500.0 0.3 0.6 0.9MemoryUnigramCE − MedianEmpirical5.05.56.06.50 1 2MemoryUnigramCE − MedianEmpirical7.607.657.707.750.0 0.1 0.2 0.3MemoryUnigramCE − MedianEmpirical6.66.87.07.20.00 0.25 0.50 0.75 1.00MemoryUnigramCE − MedianEmpirical5.56.00.0 0.5 1.0 1.5 2.0MemoryUnigramCE − MedianEmpirical5.56.00.0 0.5 1.0 1.5MemoryUnigramCE − MedianEmpirical7.47.57.60.0 0.2 0.4 0.6MemoryUnigramCE − MedianEmpirical5.255.505.750.0 0.3 0.6 0.9MemoryUnigramCE − MedianEmpirical6.06.16.26.36.46.50.0 0.3 0.6 0.9 1.2MemoryUnigramCE − MedianEmpirical6.957.007.057.107.150.0 0.1 0.2 0.3MemoryUnigramCE − MedianEmpirical6.97.07.17.20.0 0.1 0.2 0.3 0.4 0.5MemoryUnigramCE − MedianEmpirical5.05.50 5 10 15 20MemoryUnigramCE − MedianEmpiricalNorth Sami Norwegian Persian Polish Portuguese Romanian Russian Serbian Slovak Slovenian Spanish Swedish Thai Turkish Ukrainian Urdu Uyghur Vietnamese7.447.487.527.560.0 0.1 0.2 0.3MemoryUnigramCE − MedianEmpirical5.15.45.76.06.30.0 0.5 1.0 1.5 2.0MemoryUnigramCE − MedianEmpirical6.26.46.66.87.00.0 0.5 1.0MemoryUnigramCE − MedianEmpirical6.16.26.36.46.56.60.0 0.2 0.4 0.6MemoryUnigramCE − MedianEmpirical4.85.25.66.00.0 0.5 1.0 1.5 2.0MemoryUnigramCE − MedianEmpirical5.45.76.06.36.60.0 0.5 1.0 1.5 2.0MemoryUnigramCE − MedianEmpirical5.255.505.756.006.250.0 0.5 1.0 1.5MemoryUnigramCE − MedianEmpirical6.46.66.87.07.20.0 0.3 0.6 0.9MemoryUnigramCE − MedianEmpirical6.56.66.76.80.0 0.2 0.4 0.6MemoryUnigramCE − MedianEmpirical6.06.26.40.00 0.25 0.50 0.75 1.00MemoryUnigramCE − MedianEmpirical5.05.56.00 1 2MemoryUnigramCE − MedianEmpirical6.06.26.46.60.0 0.3 0.6 0.9 1.2MemoryUnigramCE − MedianEmpirical6.26.36.46.56.66.76.80.00 0.25 0.50 0.75MemoryUnigramCE − MedianEmpirical7.107.157.207.257.300.0 0.1 0.2 0.3MemoryUnigramCE − MedianEmpirical6.56.66.76.86.90.0 0.2 0.4MemoryUnigramCE − MedianEmpirical5.56.06.57.00.0 0.5 1.0 1.5MemoryUnigramCE − MedianEmpirical8.108.158.208.258.308.350.0 0.2 0.4 0.6MemoryUnigramCE − MedianEmpirical6.806.856.906.957.000.0 0.1 0.2 0.3MemoryUnigramCE − MedianEmpirical1Figure 2: Tradeoffs between memory (x axis) and surprisal (y axis) in 54 languages, for real orderings (blue) andcounterfactual baseline grammars (red). We provide 95% confidence bands for different model runs on the reallanguages, and for the median across different baseline grammars.surprisal.Theoretical Results In Theorem 1 below, wederive a bound on the memory-surprisal tradeoffcurve which can be easily estimated from corpora.Let It be the conditional mutual information be-tween words that are t steps apart, conditioned onthe intervening words:It := I[wt, w0|w1...t 1].This quantity measures how much predictive in-formation the word t steps in the past containsabout the current word.Theorem 1. Let T be a positive integer, and con-sider a listener using at mostPTt=1 t It bits ofmemory on average. Then this listener will incuraverage surprisal at least H[wt|w<t] +Pt>T It.The theorem allows us to estimate the extra sur-prisal associated with each amount of memory ca-pacity for a language. The quantities It can be esti-mated as the difference between the cross-entropyof language models that have access to the last t 1or t words. Given such estimates of It, we esti-mate tradeoff curves as in Figure 1 by tracing outT = 1, 2, . . . .Experimental Results We tested whether wordorders as found in natural language grammarsprovide efficient memory-surprisal tradeoffs. Tothis end, we compared corpora of real languagesagainst hypothetical reorderings of those lan-guages under random baseline grammars. Weused treebanks of 54 languages from UniversalDependencies 2.3 (Nivre et al., 2018).For each language, we constructed counterfac-tual word order rules by adapting the methodologyof Gildea and Temperley (2010) to Universal De-pendencies: For each syntactic relation (subject,object, ...) used in the treebank annotation, we ran-domly sampled its position relative to the head andother of siblings. For each language and each suchset of rules, we reordered the treebank accordingto these counterfactual word order rules.For each language and its counterfactuallyordered versions, we estimated the memory-surprisal tradeoff (Theorem 1) using an LSTMrecurrent neural language model, considering allintegers T = 1, . . . , 20. Hyperparameters weretuned, for each language, to minimize averagecross-entropy on counterfactual versions, intro-ducing a conservative bias against our hypothesis.Tradeoff curves are shown in Figure 2. In 50out of 54 languages, the observed orderings led tomore favorable tradeoffs than 50% of the counter-factual orderings (p < 0.0001; Exceptions: Lat-vian, North Sami, Polish, and Slovak).Taken together, our results suggest that, acrosslanguages, word order in part reflects pressures to-wards efficient online processing under memorylimitations.ReferencesThomas M. Cover and J.A. Thomas. 2006. Elements ofInformation Theory. John Wiley & Sons, Hoboken,NJ.R Futrell and R Levy. 2017. Noisy-context surprisal asa human sentence processing cost model. In EACL.E. Gibson. 1998. Linguistic complexity: locality ofsyntactic dependencies. Cognition, 68(1):1–76.D Gildea and D Temperley. 2010. Do Grammars Min-imize Dependency Length? Cognitive Science.JA Hawkins. 1994. A performance theory of order andconstituency.Roger Levy. 2008. Expectation-based syntactic com-prehension. Cognition, 106(3):1126–1177.R Lewis and S Vasishth. 2005. An activation-basedmodel of sentence processing as skilled memory re-trieval. Cognitive Science.Joakim Nivre et al. 2018. Universal dependencies 2.3.Nathaniel J. Smith and Roger Levy. 2013. The effect ofword predictability on reading time is logarithmic.Cognition, 128(3):302–319.469

Crosslinguistic Word Orders Enable an Efficient Tradeoff of Memory and Surprisal

https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1124&amp;context=scil

Crosslinguistic Word Orders Enable an Efficient Tradeoff of Memory and Surprisal

Abstract

Similar works

Full text

Available Versions

ScholarWorks@UMass Amherst