We investigate the effect of script-based (Schank and Abelson 1977) extralinguistic context on the omission of words in fragments. Our data elicited with a production task show that predictable words are more often omitted than unpredictable ones, as predicted by the Uniform Information Density (UID) hypothesis (Levy & Jaeger 2007). We take into account effects of linguistic and extralinguistic context on predictability and propose a method for estimating the surprisal of words in presence of ellipsis. Our study extends previous evidence for UID in two ways: First, we show that not only local linguistic context, but also extralinguistic context determines the likelihood of omissions. Second, we find UID effects on the omission of content words

Drenhaus, Heiner

Lemke, Robin

Reich, Ingo

Schäfer, Lisa

English

ScholarWorks@UMass Amherst

Proceedings of the Society for Computation in Linguistics Volume 3 Article 45 2020 Script Knowledge Constrains Ellipses in Fragments - Evidence from Production Data and Language Modeling Robin Lemke Saarland University, robin.lemke@uni-saarland.de Lisa Schäfer Saarland University, lisa.schaefer@uni-saarland.de Heiner Drenhaus Saarland University, drenhaus@coli.uni-saarland.de Ingo Reich Saarland University, i.reich@mx.uni-saarland.de Follow this and additional works at: https://scholarworks.umass.edu/scil  Part of the Computational Linguistics Commons, and the Psycholinguistics and Neurolinguistics Commons Recommended Citation Lemke, Robin; Schäfer, Lisa; Drenhaus, Heiner; and Reich, Ingo (2020) "Script Knowledge Constrains Ellipses in Fragments - Evidence from Production Data and Language Modeling," Proceedings of the Society for Computation in Linguistics: Vol. 3 , Article 45. DOI: https://doi.org/10.7275/mpby-zr74 Available at: https://scholarworks.umass.edu/scil/vol3/iss1/45 This Extended Abstract is brought to you for free and open access by ScholarWorks@UMass Amherst. It has been accepted for inclusion in Proceedings of the Society for Computation in Linguistics by an authorized editor of ScholarWorks@UMass Amherst. For more information, please contact scholarworks@library.umass.edu. Script knowledge constrains ellipses in fragments –Evidence from production data and language modelingRobin Lemke, Lisa Scha¨fer, Heiner Drenhaus and Ingo ReichSFB 1102, Saarland Universityrobin.lemke@uni-saarland.deAbstractWe investigate the effect of script-based(Schank and Abelson 1977) extralinguisticcontext on the omission of words in fragments.Our data elicited with a production task showthat predictable words are more often omit-ted than unpredictable ones, as predicted bythe Uniform Information Density (UID) hy-pothesis (Levy and Jaeger, 2007). We takeinto account effects of linguistic and extralin-guistic context on predictability and propose amethod for estimating the surprisal of wordsin presence of ellipsis. Our study extends pre-vious evidence for UID in two ways: First,we show that not only local linguistic context,but also extralinguistic context determines thelikelihood of omissions. Second, we find UIDeffects on the omission of content words.Background In order to communicate a mes-sage, speakers can choose between a full sen-tence (1a) and nonsentential utterances, or frag-ments (Morgan, 1973) (1b). Fragments can con-vey the same meaning as the corresponding sen-tence, but lack words that are obligatory in thesentence, like a finite verb. We investigate whypeople omit particular words in fragments and hy-pothesize that the choice between omitting and re-alizing a word is driven by the Uniform Informa-tion Density (UID) hypothesis (Levy and Jaeger,2007), which has been applied to other omissions,like that of relative pronouns (Levy and Jaeger,2007) and complementizers (Jaeger, 2010).(1) Ann and Bill are sharing a pizza. She asks:a. Would you like another slice of pizza?b. Another slice?Uniform Information Density UID states thatinformation is best distributed uniformly acrossthe utterance. Following Shannon (1949), the in-formation, or surprisal (Hale, 2001), of a word wiis defined as the negative logarithm of its likeli-hood to appear in context (2).(2) S(wi) = − log2 p (wi | context)Surprisal indexes processing effort (Hale, 2001;Levy, 2008), and a uniform distribution makes themost efficient use of the hearer’s limited cogni-tive resources. Previous research has shown thatthe optional omission of function words reflectsoptimization with respect to UID (e.g. Levy andJaeger, 2007; Jaeger, 2010). Optimization con-sists in two strategies that contribute to a uniformdistribution of information: First, omitting unin-formative words avoids inefficient local surprisalminima. Second, words that reduce the surprisalof very informative, i.e. unpredictable, followingwords are more likely to be inserted. If this rea-soning also applies to content words like pizza in(2), UID can explain why speakers sometimes usea (specific) fragment rather than a sentence: Thefragment is preferred over the sentence if it resultsfrom omitting predictable words that are obliga-tory in the corresponding full sentence.Materials and method Investigating whetheromissions are subject to UID requires (i) a set oflinguistic data containing the relevant omissionsand (ii) surprisal estimates for both the omittedand realized words in this data set. Given thesesurprisal estimates, logistic regressions can showwhether information-theoretic predictors like sur-prisal affect the likelihood of a word’s omission.Although the term context in (2) in princi-ple comprises both linguistic and extralinguis-tic context (Levy, 2008), most of the previousinformation-theoretic studies on omissions (likethe ones cited above) estimated the surprisal ofwords from corpora with n-gram language mod-els. Such models take only (part of) the linguis-tic context of the target word into account. How-441Proceedings of the Society for Computation in Linguistics (SCiL) 2020, pages 441-444.New Orleans, Louisiana, January 2-5, 2020ever, fragments often occur discourse-initially, sothat predictability depends on extralinguistic con-text that cannot be retrieved from text corpora.Therefore we collected a data set of utterancesfor tightly controlled script knowledge-based con-texts (Schank and Abelson, 1977) with a produc-tion task. This data set allows to quantify the effectof both extralinguistic and linguistic context.Subjects read a story like (3) (original materi-als in German) and produced the utterance thatthey considered most likely in that context. Sincescripts prime upcoming events (see e.g. Deloguet al., 2018), they should raise expectations aboutwhat will be said in a script-based situation. Forinstance, in (3), a request to pour the pasta into thepot or to give the speaker the pasta is probable.(3) Annika and Jenny want to cook pasta. An-nika has put a pot with water on the stove.Then she has turned the stove on. After afew minutes, the water has started to boil.Now Annika says to Jenny:In order to use empirically motivated script knowl-edge representations as stimuli, we based our ma-terials on event chains extracted from DeScript(Wanzare et al., 2016), a crowd-sourced corpus ofscript knowledge that contains about 100 descrip-tions of the stereotypical time-course of everydayactivities, such as cooking pasta. Following Man-shadi et al. (2008), we defined an event as the fi-nite verb and its nominal complement, e.g. putpot in (3). After dependency-parsing the cor-pus (Stanford parser, Klein and Manning (2003))we extracted these event representations from it.We estimated the likelihood of an event giventhe previous one with bigram language modelstrained on the manually preprocessed data for eachscript with the SRILM toolkit (Stolcke, 2002).We then extracted sequences of three events thatwere most likely to follow each other and usedthese event chains to construct our materials. Thefirst sentence in each item introduces the script(cooking pasta), and the next three ones elaboratethe event chain (put pot, turn on stove,boil water). For each of 24 items, we col-lected responses from 100 participants recruitedon the crowdsourcing platform Clickworker.Production data preprocessing As there wasa high degree of variation both between scriptsand between subjects in the data collected withthe production task, we preprocessed the data bymanually resolving pronouns and ellipses, lem-matizing the remaining words and finally pool-ing synonyms to a single lemma. Because weare interested in content words, we removed allfunction words and adverbials. Removing func-tion words is necessary because e.g. articles andprepositions cannot be freely omitted in standardGerman (Lemke, 2017; Reich, 2017) and adapta-tion to UID occurs only “within the bounds de-fined by grammar” (Jaeger, 2010, 25). Preposi-tions and distinctive case morphology were anno-tated on the noun (see (4) for an example), as thesefeatures can be important cues towards the mean-ing intended by the speaker. Adverbials were re-moved because they can remain implicit in regu-lar sentences and therefore are not involved in thegeneration of fragments (even though it might beinteresting to investigate whether this is subject toUID as well). For the utterance in (4a), prepro-cessing yields the abstract representation in (4b).(4) a. Schu¨ttepourdietheNudelnpastainindenthe.ACCTopf!potPour the pasta into the pot!b. pour pasta in.potInvestigating the effect of surprisal on omissionrequires surprisal estimates for both realized andomitted words, therefore we reconstructed all el-lipses in the original data. We added those ex-pressions that are minimally required in a full sen-tence, that is, missing verbs and/or their argu-ments. This ensures that the outcome of the in-dependent variable, surprisal, is not affected bythe dependent variable, omission. The data set foranalysis comprises a total of 2.409 sentences con-sisting in 6.816 primitive expressions (“words” inwhat follows), 1.052 (15.43%) of these words hadbeen omitted in the original data set.Surprisal estimation We investigate poten-tial effects of three measures of surprisal: (i) un-igram surprisal, (ii) context-dependent surprisalthat takes into account preceding linguistic mate-rial within the utterance and (iii) surprisal reduc-tion, i.e. how much inserting a word before a tar-get word reduces its surprisal.We estimate the unigram surprisal of each wordin the preprocessed data with unigram languagemodels with Good-Turing discount on the pre-processed data that we trained using the SRILMtoolkit (Stolcke, 2002). We trained an individuallanguage model on the data for each script sepa-442rately, because this allows to interpret surprisal asconditioned on the script-based situation, i.e. onthe extralinguistic context (5):(5) S(wi) = − log2 p(wi | contextextraling.).We use a novel method based on Hale (2001) toestimate context-dependent surprisal, that consid-ers preceding words in addition to extralinguisticcontext. The default method to quantify effects oflinguistic context on surprisal are bigram or higherorder n-gram models. However, training n-grammodels on elliptical data brings along a circularityissue observed by Levy and Jaeger (2007, 852):If predictable words are omitted more often thanunpredictable ones, their corpus frequency is notproportional to their predictability. This problemcould be addressed by ellipsis resolution, but train-ing n-gram models on the enriched data set is alsonot realistic. A trigram model trained on the en-riched data set estimates the surprisal of pot ina fragment pour pot, where pasta has beenomitted from p(pot | pour pasta). Crucially, thisis psychologically implausible, because pasta isnot included in the actual linguistic context.Therefore we estimate context-dependent sur-prisal (and surprisal reduction, see below) witha method based on the approach by Hale (2001).Hale (2001) derives surprisal from the work doneby the human parser, that consists in rejecting allparses that are compatible with the input beforebut not after processing a word. The larger the to-tal probability mass of the rejected parses is, themore informative is a word. This approach requi-res to know the likelihood of each parse, i.e. eachcomplete structure, which in our case is equivalentto its relative frequency in the enriched data set.Hale (2001) calculates the surprisal of a word wias the log ratio between the prefix probability α,i.e. the total probability mass of the parses compa-tible with an input, before and after processing wi:(6) S(wi) = logαi-1αiWe modify Hale’s approach by allowing for arbi-trarily many omissions before and after each wordin the input string in order to account for the possi-bility of ellipses when calculating a word’s effecton the set of maintained parses and consequentlyon αi. For instance, processing pour in the frag-ment pour pot rules out all parses that do notcontain pour. Processing pot now excludes allPredictors r2 t-value p-valueUnigram, context .65 70.06 < .001Unigram, reduction .48 37.99 < .001Context, reduction .62 54.0 < .001Table 1: Correlations between surprisal predictors.parses that do not contain pot somewhere afterpour, independently of whether there is a wordlike pasta between pour and pot. Surprisal iscalculated as (6) based on the prefix probabilitiesbefore and after these processing steps. Our ap-proach circumvents the circularity issue becauseit relies on nonelliptical representations. It is alsopsychologically realistic because it quantifies thework done by the parser incrementally.Finally, we calculate surprisal reduction, i.e.how much inserting wi reduces the surprisal ofwi-1, for all non-final words. For this purpose, wecalculate the ratio between the prefix probabilityat wi+1 if wi has been realized and the prefix prob-ability at wi+1 if wi has been omitted. In case ofthe example, how much the surprisal of pot is re-duced by inserting pasta is calculated as (7).(7) S reduction(pot,pasta) =αput ... potαput ... pasta ... potResults We analyzed the data with mixed effectslogistic regressions (lme4, Bates et al. (2015))that predict the omission of a word in the enricheddata set from the surprisal measures. We first con-ducted separate analyses of unigram and context-dependent surprisal on the complete data set andthen an analysis that considers both unigram sur-prisal and surprisal reduction for non-final words.In principle it would have been desirable to in-clude all three surprisal measures as predictors ina single regression analysis, but, as table 1 shows,in particular context-dependent surprisal is highlycorrelated with the other two measures.The models in the analyses of unigram sur-prisal1 and context-dependent surprisal2 containedby-script random intercepts and slopes for sur-prisal and by-subject random intercepts. In bothanalyses there are significant main effects of therespective predictor, that confirm our hypothe-sis that predictable words are more likely to beomitted. The effect for unigram surprisal (χ2 =7.39, p < .01) is stronger than that of context-1Ellipsis ∼ UnigramS + (1+UnigramS|Script) + (1|Subj)2Ellipsis ∼ ContextS + (1+ContextS|Script) + (1|Subj)443dependent surprisal (χ2 = 4.86, p < .05).The analysis that includes surprisal reduction andunigram surprisal3 was conducted on a subset ofthe data that contained those non-final words thatwere not followed by an ellipsis (55.51% of thetotal data). The final model has random interceptsfor subjects and scripts and contains significantmain effects of both predictors. The effect of un-igram surprisal (χ2 = 10.39, p < .01) replicatesthe analysis of the full data set, and the effect ofsurprisal reduction (χ2 = 27.03, p < .001) showsthat words that reduce the surprisal of the follow-ing word more strongly are more likely to be re-alized. There is no significant interaction betweenboth predictors (χ2 = 0.01, p > .9).Discussion Our study confirms the predictionsof UID on omissions in fragments: Predictablewords are more often omitted in fragments, andwords that reduce the surprisal of following onesare more often realized. This extends previous ev-idence for UID in two ways: First, we find UIDeffects on the omission of content words. Second,we show that not only local linguistic context, butalso extralinguistic context determines the likeli-hood of omissions. UID however seems not to bethe only factor in determining whether fragmentsare used, as the ratio of fragments varies even be-tween scripts with a similar mean surprisal.Our study also shows that event probabilities es-timated from a corpus of script knowledge providea reasonable model of extralinguistic context, towhich subjects adapt their lingustic behavior. Wealso propose a method for estimating by-word sur-prisal in partially elliptical data in a psychologi-cally realistic way. In our study this required adata set that we collected specfically for this pur-pose and a large amount of manual preprocessing.Future work could show inhowfar our results canbe replicated on larger and less constrained datasets when preprocessing steps like reference andellipsis resolution as well as the standardization ofthe production data are automatized.ReferencesDouglas Bates, Martin Ma¨chler, Ben Bolker, and SteveWalker. 2015. Fitting Linear Mixed-Effects Mod-els Using lme4. Journal of Statistical Software,67(1):1–48.Francesca Delogu, Heiner Drenhaus, and Matthew W.Crocker. 2018. On the predictability of event bound-3Ellipsis ∼ UnigramS * SReduction (1|Script) + (1|Subj)aries in discourse: An ERP investigation. Memory& Cognition, 46(2):315–325.John Hale. 2001. A probabilistic Earley parser as apsycholinguistic model. In Proceedings of NAACL(Vol. 2), pages 159–166.T. Florian Jaeger. 2010. Redundancy and reduc-tion: Speakers manage syntactic information den-sity. Cognitive Psychology, 61(1):23–62.Dan Klein and Christopher D. Manning. 2003. Accu-rate Unlexicalized Parsing. In Proceedings of the41st Meeting of the Association for ComputationalLinguistics, pages 423–430.Robin Lemke. 2017. Sentential or not? – An exper-imental study on the syntax of fragments. In Pro-ceedings of Linguistic Evidence 2016. University ofTu¨bingen, online publication system.Roger Levy. 2008. Expectation-based syntactic com-prehension. Cognition, 106(3):1126–1177.Roger P. Levy and T. Florian Jaeger. 2007. Speak-ers optimize information density through syntacticreduction. In Bernhard Schlo¨kopf, John Platt, andThomas Hoffman, editors, Advances in Neural In-formation Processing Systems, pages 849–856. MITPress.Mehdi Manshadi, Reid Swanson, and Andrew S Gor-don. 2008. Learning a Probabilistic Model of EventSequences from Internet Weblog Stories. In Pro-ceedings of the Twenty-First International FLAIRSConference.Jerry Morgan. 1973. Sentence fragments and thenotion ’sentence’. In Braj B. Kachru, RobertLees, Yakov Malkiel, Angelina Pietrangeli, and SolSaporta, editors, Issues in Linguistics. Papers inHonor of Henry and Rene´e Kahane, pages 719–751.University of Illionois Press, Urbana.Ingo Reich. 2017. On the omission of articles andcopulae in German newspaper headlines. Linguis-tic Variation, 17(2):186–204.Roger Schank and Robert Abelson. 1977. Scripts,Plans, Goals, and Understanding: An Enquiry intoHuman Knowledge Structures. Erlbaum, Hillsdale.Claude E. Shannon. 1949. The mathematical theory ofcommunication. In Claude E. Shannon and WarrenWeaver, editors, The Mathematical Theory of Com-munication. The University of Illinois Press, Ur-bana.Andreas Stolcke. 2002. SRILM – an extensible lan-guage modeling toolkit. In Proc. Intl. Conf. SpokenLanguage Processing, Denver, Colorado.Lilian D. A. Wanzare, Alessandra Zarcone, StefanThater, and Manfred Pinkal. 2016. DeScript: Acrowdsourced corpus for the acquisition of high-quality script knowledge. In Proceedings of LREC2016, pages 3494–3501, Portoroz, Slovenia.444

Script Knowledge Constrains Ellipses in Fragments - Evidence from Production Data and Language Modeling

https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1125&context=scil

Script Knowledge Constrains Ellipses in Fragments - Evidence from Production Data and Language Modeling

Abstract

Similar works

Full text

Available Versions

ScholarWorks@UMass Amherst