- Numerous studies of utterance mediated gaze in visual scenes have demonstrated that sentence processing is not only incremental but also eager: During processing, listeners form expectations about upcoming arguments and make anticipatory eye movements to relevant displayed objects. - In particular, selectional information from verbs has been shown to guide visual attention to appropriate objects; upon hearing “the boy will eat”, listeners start looking at edible objects even before they are mentioned [1, 2]. - While these studies provide valuable insights into semantic processing, it is not clear whether anticipatory eye movements indeed reflect the purely linguistic activation of likely arguments or whether these anticipatory processes are influenced by the circumscribed visual context. - We present a German cross-modal priming experiment in which we examined listeners sensitivity to selectional restrictions between verbs and their object arguments

Crocker, M.

Weber, A.

English

MPG.PuRe

 Numerous studies of utterance mediated gaze in visual scenes have demonstrated thatsentence processing is not only incremental but also eager: During processing,listeners form expectations about upcoming arguments and make anticipatory eyemovements to relevant displayed objects. In particular, selectional information from verbs has been shown to guide visualattention to appropriate objects; upon hearing “the boy will eat”, listeners start lookingat edible objects even before they are mentioned [1, 2]. While these studies provide valuable insights into semantic processing, it is not clearwhether anticipatory eye movements indeed reflect the purely linguistic activation oflikely arguments or whether these anticipatory processes are influenced by thecircumscribed visual context. We present a German cross-modal priming experiment in which we examined listenerssensitivity to selectional restrictions between verbs and their object arguments. Nu erous studies of utterance ediated gaze in visual scenes have de onstrated thatsentence processing is not only incre ental but also eager: During processing,listeners for  expectations about upco ing argu ents and ake anticipatory eyeove ents to relevant displayed objects. In particular, selectional infor ation fro  verbs has been shown to guide visualattention to appropriate objects; upon hearing “the boy will eat”, listeners start lookingat edible objects even before they are entioned [1, 2]. hile these studies provide valuable insights into se antic processing, it is not clearwhether anticipatory eye ove ents indeed reflect the purely linguistic activation oflikely argu ents or whether these anticipatory processes are influenced by thecircu scribed visual context. e present a er an cross- odal pri ing experi ent in which we exa ined listenerssensitivity to selectional restrictions between verbs and their object argu ents.INTRODUCTION30 German auditory sentence onsets consisting of a subject NP (the agent) followed bya restrictive verb: “Die Frau bäckt” (‘The woman bakes’)Displays with the agent (Frau, ‘woman’), an object either semantically appropriate asargument for the verb (Torte, ‘cake’) or inappropriate (Tanne, ‘pine’), and two distractorobjects.Visual lexical decision items either semantically appropriate (Pizza, ‘pizza’) orinappropriate (Palme, ‘palm tree) as arguments for the verb. (Always different fromdepicted arguments.)58 filler trials with varying syntactic structures, restrictive and unrestrictive verbs, cutoffs at different locations, or full sentences; with lexical decisions to inappropriatearguments, appropriate arguments, arguments actually mentioned in sentences, ornonwords.Ratio of word-to-nonword in lexical decision:  50%.Both the appropriate visual argument (pie) and the appropriate lexical decision item(pizza) were highly plausible arguments for sentence onsets (rating study).Presentation of items was counterbalanced such that appropriate pictures and lexicaldecision items also appeared as inappropriate items (“Die Frau pflanzt”, ‘The womanplants’).30 er an auditory sentence onsets consisting of a subject NP (the agent) followed bya restrictive verb: “Die Frau bäckt” (‘The wo an bakes’)Displays with the agent (Frau, ‘wo an’), an object either se antically appropriate asargu ent for the verb (Torte, ‘cake’) or inappropriate (Tanne, ‘pine’), and two distractorobjects.Visual lexical decision ite s either se antically appropriate (Pizza, ‘pizza’) orinappropriate (Pal e, ‘pal  tree) as argu ents for the verb. (Always different frodepicted argu ents.)58 filler trials with varying syntactic structures, restrictive and unrestrictive verbs, cutoffs at different locations, or full sentences; with lexical decisions to inappropriateargu ents, appropriate argu ents, argu ents actually entioned in sentences, ornonwords.Ratio of word-to-nonword in lexical decision:  50 .Both the appropriate visual argu ent (pie) and the appropriate lexical decision ite(pizza) were highly plausible argu ents for sentence onsets (rating study).Presentation of ite s was counterbalanced such that appropriate pictures and lexicaldecision ite s also appeared as inappropriate ite s (“Die Frau pflanzt”, ‘The wo anplants’).METHODThe influence of the scene on linguistic expectations:Evidence from cross-modal priming in visual worlds i l      li i i  ii   l i i  i  i l l                                              Andrea Weber & Matthew W. Crocker, Saarland University    aweber@coli.uni-sb.de Facilitated lexical decision times for appropriate items in Experiment 1, regardless ofthe scene, provide evidence for purely linguistic anticipation, confirming the gatedcompletion findings of Altmann (1999). In contrast to previous priming results (see e.g. [3]), however, appropriate pictureprimes slowed down lexical decision times. We suggest, that visually attending thepicture of an object based on supporting auditory input, set up expectations for thatspecific object to follow as verb argument; when the expectations were not met, lexicaldecision times were slowed. To test whether indeed the combination of auditory primes with picture primes wasresponsible for the semantic interference, we are currently re-running the study withjust picture primes. Also the presence of four picture primes rather than a single pictureprime might have influenced the result patterns. (Indeed, lexical decision times inExperiment 2 were always faster than in Experiment 1.) We will therefore comparelexical decision times following single picture primes and multiple picture primes. Facilitated lexical decision ti es for appropriate ite s in Experi ent 1, regardless ofthe scene, provide evidence for purely linguistic anticipation, confir ing the gatedco pletion findings of Alt ann (1999). In contrast to previous pri ing results (see e.g. [3]), however, appropriate picturepri es slowed down lexical decision ti es. e suggest, that visually attending thepicture of an object based on supporting auditory input, set up expectations for thatspecific object to follow as verb argu ent; when the expectations were not et, lexicaldecision ti es were slowed. To test whether indeed the co bination of auditory pri es with picture pri es wasresponsible for the se antic interference, we are currently re-running the study withjust picture pri es. Also the presence of four picture pri es rather than a single picturepri e ight have influenced the result patterns. (Indeed, lexical decision ti es inExperi ent 2 were always faster than in Experi ent 1.) e will therefore co parelexical decision ti es following single picture pri es and ultiple picture pri es.GENERAL DISCUSSIONREFERENCES[1] Altmann, G.T.M., & Kamide, Y. (1999). Incremental interpretation at verbs: Restricting the domainof subsequent reference. Cognition, 72.[2] Kamide,Y., Altmann,G.T.M., & Haywood,S.(2003).The time course of prediction in incremental sentence processing: Evidence from anticipatory eye-movements. Journal of Memory and Language, 49.[3] McQueen, J. & Huettig, F. (2005). Semantic and phonological priming of auditory lexical decisionby pictures and printed words. AMLaP, Ghent, Belgium.[4] Altmann, G.T.M. (1999). Thematic role assignment in context. Journal of Memory and Language,41.REFERENCES[1] Altmann, G.T.M., & Kamide, Y. (1999). Incremental interpretation at verbs: Restricting the domainof subsequent reference. Cognition, 72.[2] Kamide,Y., Altmann,G.T.M., & Haywood,S.(2003).The time course of prediction in incremental sentence processing: Evidence from anticipatory eye-movements. Journal of Memory and Language, 49.[3] McQueen, J. & Huettig, F. (2005). Semantic and phonological priming of auditory lexical decisionby pictures and printed words. AMLaP, Ghent, Belgium.[4] Altmann, G.T.M. (1999). Thematic role assignment in context. Journal of Memory and Language,41. Do anticipatory eye movements in visual-world studies indeed reflect anticipation oflinguistic items in everyday sentence processing or are they merely driven by thepresence of depicted objects? Do anticipatory eye ove ents in visual-world studies indeed reflect anticipation oflinguistic ite s in everyday sentence processing or are they erely driven by thepresence of depicted objects?QUESTION48 German listenersTask: listen to sentences while seeing pictures and respond to subsequent visuallexical decision itemsAuditory: The woman bakesVisually: pie or pineLexical decision: pizza or palm treeAs expected, reaction times were faster for lexical decision items which weresemantically appropriate (pizza) than for inappropriate items (palm tree; F1[1, 47] =8.54, p < .01; F2[1, 29] = 4.24, p < .05).Surprisingly, however, reaction times were slowed when the display included a pictureof an appropriate argument (pie) prior to lexical decision compared to when noappropriate argument was depicted (pine). This effect was significant by subjects (F1[1,47] = 5.03, p = .03; F2[1, 29] = 2.04 p > .05).The semantic interference from pictures occurred both when the lexical decision itemwas appropriate and when it was not (no interaction: F1 & F2 < 1).48 er an listenersTask: listen to sentences while seeing pictures and respond to subsequent visuallexical decision ite sAuditory: The wo an bakesVisually: pie or pineLexical decision: pizza or pal  treeAs expected, reaction ti es were faster for lexical decision ite s which werese antically appropriate (pizza) than for inappropriate ite s (pal  tree; F1[1, 47] =8.54, p < .01; F2[1, 29] = 4.24, p < .05).Surprisingly, however, reaction ti es were slowed when the display included a pictureof an appropriate argu ent (pie) prior to lexical decision co pared to when noappropriate argu ent was depicted (pine). This effect was significant by subjects (F1[1,47] = 5.03, p = .03; F2[1, 29] = 2.04 p > .05).The se antic interference fro  pictures occurred both when the lexical decision itewas appropriate and when it was not (no interaction: F1 & F2 < 1).EXPERIMENT 1To determine the effect of additional picture primes on lexical decision times, onlyauditory primes were included in Experiment 2.24 German listenersTask: listen to sentences and respond to subsequent visual lexical decision items (nopictorial prime)Auditory: The woman bakesLexical decision: pizza or palm treeAs in Experiment 1, reaction times were fast for lexical decision items which weresemantically appropriate (pizza) than for inappropriate items (palm tree; F1[1, 23] =6.95, p < .02; F2[1, 29] = 3.99, p = .05).The presence of picture primes in Experiment 1, resulted in significantly slower reactiontimes compared to Experiment 2, both when the picture primes were appropriate orinappropriate.To deter ine the effect of additional picture pri es on lexical decision ti es, onlyauditory pri es were included in Experi ent 2.24 er an listenersTask: listen to sentences and respond to subsequent visual lexical decision ite s (nopictorial pri e)Auditory: The wo an bakesLexical decision: pizza or pal  treeAs in Experi ent 1, reaction ti es were fast for lexical decision ite s which werese antically appropriate (pizza) than for inappropriate ite s (pal  tree; F1[1, 23] =6.95, p < .02; F2[1, 29] = 3.99, p = .05).The presence of picture pri es in Experi ent 1, resulted in significantly slower reactionti es co pared to Experi ent 2, both when the picture pri es were appropriate orinappropriate.EXPERIMENT 2650670690710730750770appr. LD inappr. LDappr. pict.inappr. pict.550570590610630650670appr. LD inappr. LDDie Frau bäckt2000 ms400 msPizza

The influence of the scene on linguistic expectations: Evidence from cross-model priming in visual worlds

http://hdl.handle.net/11858/00-001M-0000-0013-2F3C-F

The influence of the scene on linguistic expectations: Evidence from cross-model priming in visual worlds

Abstract

Similar works

Full text

Available Versions

MPG.PuRe

MPG.PuRe