Visual constraints modulate stereotypical predictability of agents during situated language comprehension

Rodriguez A, Burigo M, Knoeferle P. Visual constraints modulate stereotypical predictability of agents during situated language comprehension. In: Proceedings of the 38th Annual Cognitive Science Society Meeting. 2016

Situated Sentence Processing: The Coordinated Interplay Account and a Neurobehavioral Model

Crocker MW, Knoeferle P, Mayberry M. Situated Sentence Processing: The Coordinated Interplay Account and a Neurobehavioral Model. Brain and Language. 2010;112(3):189-201

The influence of prior visual gender and action cues versus long-term knowledge in (situated) language processing

Rodríguez A. The influence of prior visual gender and action cues versus long-term knowledge in (situated) language processing. Bielefeld: Universität Bielefeld; 2018.Studies on situated language comprehension (i.e., comprehension in rich visual contexts), have shown that the comprehender makes use of different information sources in order to establish visual reference and to visually anticipate entities in a scene while understanding language (reflecting expectations on what might be mentioned next). Semantics and world-knowledge (i.e., experiential, long-term knowledge) are among these sources. For instance, when listening to a sentence like *The girl will ride...*, the comprehender will likely anticipate an object that a girl could ride, e.g., a carrousel, rather than other objects, such as a motorbike (Kamide, Altmann, & Haywood, 2003). However, following the inspection of events (featuring agents acting upon objects or patients), comprehenders have so far shown a preference to visually anticipate the agents or objects that have been seen as part of those prior events (i.e., *recent-event preference* or the preference for event-based representations; Abashidze, Carminati, & Knoeferle, 2014; Knoeferle, Carminati, Abashidze, & Essig, 2011). This preference emerged even when other plausible objects or better stereotypically fitting agents were present. Although the preference for event-based information over other sources (e.g., plausibility or stereotypicality) seems to be strong and has been accommodated in accounts of situated language comprehension (Knoeferle & Crocker, 2006, 2007), its nature when comprehenders generate expectations is still unspecified. Crucially, the preference for recent events has not been generalized from action events to other types of information in the visual and linguistic contexts. To further examine this issue, this thesis investigated the role of a particular type of information during situated language comprehension under the influence of prior events, namely, visual gender and action cues and knowledge about gender stereotypes. As many studies in the field of psycholinguistics have highlighted, gender (both a biological and a social feature of human beings) is relevant in language comprehension (e.g., grammatical gender can serve to track reference in discourse, and gender-stereotype knowledge can bias our interpretation of a sentence). However, little psycholinguistic research has examined the comprehension of gender information in a visual context. We argue that gender is worth exploring in a paradigm where prior event representations can be pitted against long-term knowledge. Not only that, inspired by experiments using mismatch designs, we wanted to see how the visual attention of the comprehender might be affected as a function of referential incongruencies (i.e., mismatches between visual events and linguistic information, e.g., Knoeferle, Urbach, & Kutas, 2014; Vissers, Kolk, Van de Meerendonk, & Chwilla, 2008; Wassenaar & Hagoort, 2007) and incongruences at the level of worldknowledge (i.e., gender stereotypes; e.g., Duffy & Keir, 2004; Kreiner, Sturt, & Garrod, 2008). By doing so, we could get insights into how both types of sources (event-based information and gender-stereotype knowledge from language) are used, i.e., whether one is more important than the other or if both are equally exploited in situated language comprehension. We conducted three eye-tracking, visual-world experiments and one EEG experiment. In all of these experiments, participants saw events taking place prior to sentence comprehension, i.e., videos of (female or male) hands acting upon objects. In the eye-tracking experiments, following the videos, a visual scene appeared with the faces of two potential agents: one male and one female1. While the agent matching the gender features from prior events (i.e., the hands) was considered as the target agent, the other potential agent, whose gender was not cued in previous events, was the competitor agent. The visual scene in Experiment 3 further included the images of two objects; one was the target object (i.e., the object that appeared in prior events), while the other was a competitor object with opposite stereotypical valence. During the presentation of this scene, an OVS sentence was presented (e.g., translation from German: ‘The cakeNP1/obj bakesV soonADV SusannaNP2/subj’). We used the non-canonical OVS word order as opposed to SVO (more commonly used in prior research, e.g., Knoeferle, Carminati, et al., 2011) precisely to examine participants’ expectations towards the agent, who was mentioned at final position. We manipulated two factors. One factor was the match between prior visual events and language: there were action-verb(-phrase) mismatches in Experiments 1 and 3, and mismatches between the gender of the hands and the final subject (i.e., the proper name) in Experiments 2 and 4. The second manipulation, present in Experiments 1 to 3, was the match between the stereotypical valence of the described actions/events in the sentence and the target agent’s gender. In the eye-tracking experiments, we measured participants’ visual attention towards the agents’ faces during sentence comprehension. In the EEG experiment, we measured ERP responses time-locked to the final, proper name region (i.e., Susanna). Participants’ task was to verify via button press whether the sentence matched the events they just saw. In line with prior research, our results support the idea that the preference for eventbased representations generalizes to another cue, i.e., gender features from the hands of an agent during prior events. Participants generally preferred to look at the target agent compared to the competitor. These results also suggest that the recent-event preference does not just rely on representations of full objects, agents and events, but also subtler (gender) features that serve to identify feature-matching targets during comprehension (i.e., faces of agents are inspected based on the gender features from hands seen in prior events). This preference is however modulated by mismatches in language, i.e., whenever the actions described or the gender implied by the final noun in the sentence were at odds with prior events, attention towards the target agent was reduced. In addition, the scene configuration of Experiment 3 gave rise to gender stereotypicality effects, which had not yet been found in prior studies using a similar design. Participants looked at the target agent (vs. the competitor) to a greater extent when the action described by the sentence stereotypically matched (vs. mismatched) them. As for the electrophysiological response towards mismatches between event-based gender cues and language, we found a biphasic ERP response, which suggests that this type of verification requires two semantically-induced stages of processing. This response had commonalities both with some effects found in strictly linguistic/discourse contexts but also with previously observed mismatch effects in picture-sentence verification studies (i.e., role relation and action mismatches; Knoeferle et al., 2014), which suggests that a similar (perhaps a single) processing mechanism might be involved in several visuolinguistic relations. In sum, our results using gender and action cues from prior events and long-term knowledge call for a more refined consideration of the different aspects involved in (situated) language comprehension. On the one hand, existing accounts need to accommodate further reconciliations/verifications of visuolinguistic relations (e.g., roles, actions, gender features, etc.). When it comes to listeners generating expectations during comprehension while inspecting the visual world, we further suggest that a weighted system (i.e., a system indexing the strength of the expectation and how different information sources contribute to it; also suggested in Münster, 2016), applies for gender of information. Not only event-based representations, but also different discrepancies between these representations and language and, depending on the concurrent visual scene configuration, long-term knowledge (e.g., pertaining to gender stereotypes), can affect weighted expectations. Biosocial aspects such as gender may be of particular interest to answer some of the open questions in how situated language comprehension works, as these aspects can be found and manipulated at different levels of communication (e.g., the comprehender, the speaker, the linguistic content, etc.)

Autistic adults anticipate and integrate meaning based on the speaker’s voice: Evidence from eye-tracking and event-related potentials

Typically developing (TD) individuals rapidly integrate information about a speaker and their intended meaning while processing sentences online. We examined whether the same processes are activated in autistic adults, and tested their timecourse in two pre-registered experiments. Experiment 1 employed the visual world paradigm. Participants listened to sentences where the speaker’s voice and message were either consistent or inconsistent (e.g. “When we go shopping, I usually look for my favourite wine”, spoken by an adult or a child), and concurrently viewed visual scenes including consistent and inconsistent objects (e.g. wine and sweets). All participants were slower to select the mentioned object in the inconsistent condition. Importantly, eye movements showed a visual bias towards the voiceconsistent object, well before hearing the disambiguating word, showing that autistic adults rapidly use the speaker’s voice to anticipate the intended meaning. However, this target bias emerged earlier in the TD group compared to the autism group (2240ms vs 1800ms before disambiguation). Experiment 2 recorded ERPs to explore speaker-meaning integration processes. Participants listened to sentences as described above, and ERPs were time-locked to the onset of the target word. A control condition included a semantic anomaly. Results revealed an enhanced N400 for inconsistent speaker-meaning sentences that was comparable to that elicited by anomalous sentences, in both groups. Overall, contrary to research that has characterised autism in terms of a local processing bias and pragmatic dysfunction, autistic people were unimpaired at integrating multiple modalities of linguistic information, and were comparably sensitive to speaker-meaning inconsistency effects

The Processing of Emotional Sentences by Young and Older Adults: A Visual World Eye-movement Study

Carminati MN, Knoeferle P. The Processing of Emotional Sentences by Young and Older Adults: A Visual World Eye-movement Study. Presented at the Architectures and Mechanisms of Language and Processing (AMLaP), Riva del Garda, Italy

Causal inference: relating language to event representations and events in the world

Events are not isolated but rather linked to one another in various dimensions. In language processing, various sources of information—including real-world knowledge, (representations of) current linguistic input and non-linguistic visual context—help establish causal connections between events. In this review, we discuss causal inference in relation to events and event knowledge as one aspect of world knowledge, and their representations in language comprehension. To evaluate the mechanism and time course of causal inference, we gather insights from studies on (1) implicit causality/consequentiality as a specific form of causal inference regarding the protagonists of cause/consequence events, and (2) the processing of causal relations. We highlight the importance of methodology in measuring causal inference, compare the results from different research methods, and emphasize the contribution of the visual-world paradigm to achieve a better understanding of causal inference. We recommend that further investigations of causal inference consider temporally sensitive measures and more detailed contexts

The Effects of Social Context and Perspective on Language Processing: Evidence from Autism Spectrum Disorder

This thesis aimed to provide new insights into the role of perspective and non-linguistic context in language processing among autistic and typically developing (TD) adults. The mental simulation account and the one-step model state that language is mentally simulated and interpreted in context, suggesting that these processes are activated online while linguistic input is processed. Little is known of whether the same processes are activated in autism. In seven experiments (four were fully pre-registered), I used offline and online measures (e.g. EEG, eye-tracking) to investigate how social factors, such as the perspective, speaker's voice, emotional states of the characters, and topic of conversation influence language comprehension in both lab and real-life settings, in autism and TD adults. Based on the weak central coherence (WCC), and the complex information processing disorder (CIPD) theories, it was expected that autistic adults would struggle to integrate the social context with language, or at least show some subtle delays in the time-course of these anticipation/integration processes. First, I failed to find the same effect as previous findings, showing enhanced processing for personalized language, suggesting that this process is dependent on individual preferences in perspective-taking and task demands. Furthermore, I found that contrary to the WCC, autistic individuals had an intact ability to integrate social context online, while extracting the meaning from language. There were subtle differences in the time-course and strength of these processes between autistic and TD adults under high cognitive load. Findings are in line with CIPD hypothesis, showing that online language processes are disrupted as task demands increase, which consequently affect the quality of their social interactions. Future research should further investigate how these subtle differences impact social communication abilities in everyday life in autism

Linguistic variation from cognitive variability: the case of English \u27have\u27

In this dissertation, I seek to construct a model of meaning variation built upon variability in linguistic structure, conceptual structure, and cognitive makeup, and in doing so, exemplify an approach to studying meaning that is both linguistically principled and neuropsychologically grounded. As my test case, I make use of the English lexical item \u27have\u27 by proposing a novel analysis of its meaning based on its well-described variability in English and its embedding into crosslinguistically consistent patterns of variation and change. I support this analysis by investigating its real-time comprehension patterns through behavioral, electropsychophysiological, and hemodynamic brain data, thereby incorporating dimensions of domain-general cognitive variability as crucial determinants of linguistic variability. Per my account, \u27have\u27 retrieves a generalized relational meaning which can give rise to a conceptually constrained range of readings, depending on the degree of causality perceived from either linguistic or contextual cues. Results show that comprehenders can make use of both for \u27have\u27-sentences, though they vary in the degree to which they rely on each. At the very broadest level, the findings support a model in which the semantic distribution of \u27have\u27 is inherently principled due to a unified conceptual structure. This underlying conceptual structure and relevant context cooperate in guiding comprehension by modulating the salience of potential readings, as comprehension unfolds; though, this ability to use relevant context--context-sensitivity--is variable but systematic across comprehenders. These linguistic and cognitive factors together form the core of normal language processing and, with a gradient conceptual framework, the minimal infrastructure for meaning variation and change

Linguistic Variation from Cognitive Variability: The Case of English \u27Have\u27

In this dissertation, I seek to construct a model of meaning variation built upon variability in linguistic structure, conceptual structure, and cognitive makeup, and in doing so, exemplify an approach to studying meaning that is both linguistically principled and neuropsychologically grounded. As my test case, I make use of the English lexical item ‘have\u27 by proposing a novel analysis of its meaning based on its well-described variability in English and its embed- ding into crosslinguistically consistent patterns of variation and change.I support this analysis by investigating its real-time comprehension patterns through behavioral, electropsychophysiological, and hemodynamic brain data, thereby incorporating dimensions of domain-general cognitive variability as crucial determinants of linguistic variability. Per my account, ‘have\u27 retrieves a generalized relational meaning which can give rise to a conceptually constrained range of readings, depending on the degree of causality perceived from either linguistic or contextual cues. Results show that comprehenders can make use of both for ‘have\u27-sentences, though they vary in the degree to which they rely on each.At the very broadest level, the findings support a model in which the semantic distribution of ‘have\u27 is inherently principled due to a unified conceptual structure. This underlying conceptual structure and relevant context cooperate in guiding comprehension by modulating the salience of potential readings, as comprehension unfolds; though, this ability to use relevant context–context-sensitivity–is variable but systematic across comprehenders. These linguistic and cognitive factors together form the core of normal language processing and, with a gradient conceptual framework, the minimal infrastructure for meaning variation and change

Integrating pragmatic reasoning in an efficiency-based theory of utterance choice

This thesis explores new methods of accounting for discourse-level linguistic phenomena, using computational modeling. When communicating, efficient speakers frequently choose to either omit, or otherwise reduce the length of their utterances wherever possible. Frameworks such as Uniform Information Density (UID) have argued that speakers preferentially reduce or omit those elements that are more predictable in context, and easier to recover. However, these frameworks have nothing to say about the effects of a linguistic choice on how a message is interpreted. I run 3 experiments which show that while UID posits no specific consequences to being "overinformative" (including more information in an utterance than is necessary), in fact overinformativeness can trigger pragmatic inferences which alter comprehenders' background beliefs about the world. In this case, I show that the Rational Speech Act (RSA) model, which models back-and-forth pragmatic reasoning between speakers and comprehenders, predicts both efficiency-based utterance choices, as well as any consequent change in perceived meaning. I also provide evidence that it's critical to model communication as a lossy process (which UID assumes), which allows the RSA model to account for phenomena that it otherwise is not able to. I further show that while UID predicts increased use of pronouns when referring to more contextually predictable referents, existing research does not unequivocally support this. I run 2 experiments which fail to show evidence that speakers use reduced expressions for predictable elements. In contrast to UID and similar frameworks, the RSA model can straightforwardly predict the results that have been observed to date. In the end, I argue that the RSA model is a highly attractive alternative for modeling speaker utterance choice at the discourse level. When it reflects communication as a lossy process, it is able to predict the same predictability-driven utterance reduction that UID does. However, by additionally modeling back-and-forth pragmatic reasoning, it successfully models utterance choice phenomena that simpler frameworks cannot account for.Diese Arbeit erforscht neue Methoden, linguistische Phänomene auf Gesprächsebene per Computermodellierung zu erfassen. Effiziente Sprecher:innen entscheiden sich bei der Kommunikation häufig dazu, wenn immer es möglich ist, Äußerungen entweder ganz auszulassen oder aber ihre Länge zu reduzieren. Modelle wie Uniform Information Density (UID) argumentieren, dass Sprecher:innen vorzugsweise diejenigen Elemente auslassen, die im jeweiligen Kontext vorhersagbarer und einfacher wiederherzustellen sind. Allerdings sagen diese Modelle nichts über die Auswirkungen einer linguistischen Entscheidung bezüglich der Interpretation einer Nachricht aus. Ich führe drei Experimente durch, die zeigen, dass wenngleich UID keine spezifischen Auswirkungen von "Überinformation" (einer Äußerung mehr Information als nötig geben) postuliert, Überinformationen doch pragmatische Schlussfolgerungen, die das gedankliche Weltmodell der Versteher:innen ändern können, auslöst. Für diesen Fall zeige ich, dass das Rational-Speech-Act-Modell (RSA), welches pragmatische Hin-und-Her-Schlussfolgerungen zwischen Sprecher:innen und Versteher:innen modelliert, sowohl effizienzbasierte Äußerungsauswahl als auch jegliche resultierende Verständnisänderung vorhersagt. Ich liefere auch Anhaltspunkte dafür, dass es entscheidend ist, Kommunikation als verlustbehafteten Prozess zu modellieren (wovon UID ausgeht), was es dem RSA-Modell erlaubt, Phänomene einzubeziehen, wozu es sonst nicht in der Lage wäre. Weiterhin zeige ich, dass obschon UID beim Bezug auf kontextuell vorhersagbarere Bezugswörter eine erhöhte Nutzung von Pronomen vorhersagt, dies von existierender Forschung nicht einstimmig gestützt wird. Ich führe zwei Experimente durch, die keine Anhaltspunkte dafür, dass Sprecher:innen reduzierte Ausdrücke für vorhersagbare Elemente verwenden, finden. Im Gegensatz zu UID und ähnlichen Modellen kann dass RSA-Modell direkt die bislang beobachteten Resultate vorhersagen. Schließlich lege ich dar, warum das RSA-Modell eine höchst attraktive Alternative zur Modellierung von Sprachäußerungsentscheidungen auf Gesprächsebene ist. Wenn es Kommunikation als einen verlustbehafteten Prozess widerspiegelt, kann es dieselbe vorhersagebasierte Äußerungsreduktion vorhersagen wie auch UID. Modelliert man jedoch zusätzlich pragmatische Hin-und-Her-Schlussfolgerungen, modelliert RSA erfolgreich Phänomene bei Äußerungsentscheidungen, die einfachere Modelle nicht abbilden können.German Research Foundation (DFG) as part of SFB 1102 - Information Density and Linguistic Encoding (IDeaL