Clarke, Alasdair

Elsner, Micha

Rohde, Hannah

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      Influence of Visual Complexity on referring ExpressionGenerationCitation for published version:Rohde, H, Clarke, A & Elsner, M 2014, 'Influence of Visual Complexity on referring Expression Generation'27th Annual CUNY Conference on Human Sentence Processing, Columbus, United States, 13/03/14, pp. 1.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Publisher's PDF, also known as Version of recordPublisher Rights Statement:© Rohde, H. (2014). Influence of Visual Complexity on referring Expression Generation. 1. Poster sessionpresented at 27th Annual CUNY Conference on Human Sentence Processing, Columbus, United States.General rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 05. Apr. 20191. Abstract Inﬂuence of Visual Complexity on Referring Expression Genera:on  Hannah Rohde, Alasdair Clarke, & Micha Elsner The 27th CUNY Conference on Human Sentence Processing, 13‐15 March 2014, OSU ReferencesClarke, A., Elsner, M. & Rohde, H. (2013). Where's Wally: the influence of visual salience on referring expression generation. Frontiers in Perception Science: Special Issue on Scene Understanding, 4(329).Coco, M. I., & Keller, F. (2012). Scan pattern predicts sentence production in the cross-modal processing of visual scenes. Cognitive Science, 36.Dale, R., & Reiter, E. (1995). Computational interpretations of the Gricean maxims in the generation of referring expressions. Cognitive Science, 19.Pechmann, T. (2009). Incremental speech production and referential overspecification. Linguistics, 27(1).Spivey, M., Tyler, M., Eberhard, K & Tanenhaus, M. Linguistically mediated visual search. Psychological Science, 12(4).Viethen, J., & Dale, R. (2008). The use of spatial relations in referring expressions. Proceedings of INLG, 5.Contact:  Hannah.Rohde@ed.ac.uk2. Modeling Referring Expression Generation 3. A Role for Visual Perception? 6. Production Experiment 8.  Conclusion 4. Hypothesis from visual search Incremental Algorithm (Pechmann, 1989; Dale & Reiter, 1995) Goal:  account for target overspecification in REG Visual search research shows:  Finding a target is slower with more distractors, but only with heterogeneous distractors.Question:  Does this extend from perception to production?Full-stop prompt: IC verbs yield more Explanation continuations than Non-IC-  Just as the Incremental Algorithm posits incrementality for feature inclusion in the referring expression, our results suggest that feature extraction/checking is likewise an active and ongoing process.-  To scale well, REG models should incorporate cues like scene complexity and be informed by findings from visual perception.Naïve assumptions: (i) "the set of candidate features can be identified easily"        Do certain visual contexts impede feature idenfitication?7.  Results Total speaking time:        In describing a target among distractors, speakers must extract relevant properties of a visual scene and formulate a coherent referring expression (RE) to pick out that target. Although linguistic cues influence how we see the world (Spivey et al., 2001) and properties of our visual system influence what we choose to say (Coco & Keller, 2012), models of referring expression generation (REG) have largely sidestepped a role for vision (Clarke et al., 2013). We ask how the assumptions of a well-known model, the Incremental Algorithm, hold up in visually complex scenes.  Our findings suggest that, just as the Incremental Algorithm posits incrementality for feature inclusion in the RE, feature extraction/checking is likewise an active and ongoing process. To scale well, REG models should incorporate cues like scene complexity and be informed by findings from visual perception.IA procedure:       - given a set of target features (e.g., color, size, shape)         - for each feature         - for each distractor                 - add feature to RE if distractor is excluded      - terminate when RE is unambiguous            - speech may start before terminationSpeaker says:“the small blue ball”“the small ball”even though disambiguated by:(ii) "distractor checking is always a serial process”       Can speakers pre-attentively perceive feature effectiveness?Possible impact of visual context:   Heterogeneous distractors      Feature identification is hard?   Many similar distractors     Single feature (e.g. shape) is enough?Perception-Informed Hypothesis: If visual search  findings extend to REG feature extraction/checking…Then speaker production will be slowed by large distractor N, via an interaction with Heterogeneity.Incremental Algorithm Hypothesis:  IA has noprediction for an N  x  Distractor Heterogeneityinteraction because the role for vision is left unspecified.Instructions:  “describe the target so that a listener could  quickly and accurately find that shape in the same grid”Participants:  18 English speakers from University of EdinburghCoding:     - speech onset     - speech offset     - inclusion of relative descriptionsAlso have eyetracking; transcription and analysis ongoing!Confirmed hypothesis from visual search:          - Little effect of N in easy, medium (+5ms/obj)        - Large effect of N with most heterogeneity (+32ms/obj)681025 50 75 100 125N2totalcolour.distconstantskeweduniformEffects of N (p<.001), heterogeneity (p<.05), driven by predicted  interaction (p<.001).Onset to speech time:Interaction only in post-onset speech suggests full feature extraction is non-trivial and is completed after speech onsetOnly observed effect is small effect of N on onsets (p<.001;  +2ms/object).1.001.251.501.7525 50 75 100 125N2onsetPlusHesitationcolour.distconstantskeweduniformSource of post-onset delays:-  Speakers express same information more slowly (disfluency)  - Provide more information (taking longer to communicate)No difference in use of relative descriptions  for N or HeterogeneityVariation likely reflects patterns of disfluencyImage: Viethen and Dale 20085. Stimuli - 60 grids of colored shapes that varied…    - set size N (25, 49, 81, 121)    - heterogeneity (=difficulty of visual search)Target always disambiguated by shape alone (“square”)Fillers:  60 grids with non-unique targets which required relative         descriptions (“leftmost red circle”)easy medium    hard        …However, overspecification does not vary by conditioneasy hard mediu  easy hard medium 

Influence of Visual Complexity on referring Expression Generation

https://www.pure.ed.ac.uk/ws/files/18515827/RohdeClarkeElsner.cuny.2014.pdf

Influence of Visual Complexity on referring Expression Generation

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer