Visual gender cues elicit agent expectations: different mismatches in situated language comprehension

Abstract Previous research has shown that visual cues (depicted events) can have a strong effect on language comprehension and guide attention more than stereotypical thematic role knowledge ('depicted / recent event preference'). We examined to which extent this finding generalizes to another visual cue (gender from the hands of an agent) and to which extent it is modulated by picture-sentence incongruence. Participants inspected videos of hands performing an action, and then listened to non-canonical German OVS sentences while we monitored their eye gaze to the faces of two potential subjects / agents (one male and one female). In Experiment 1, the sentential verb phrase matched (vs. mismatched) the video action and in Experiment 2, the sentential subject matched (vs. mismatched) the gender of the agent's hands in the video. Additionally, both experiments manipulated gender stereotypicality congruence (i.e. whether the gender stereotypicality of the described actions matched or mismatched the gender of the hands in the video). Participants overall preferred to inspect the target agent face (i.e. the face whose gender matched that of the hands seen in the previous video), suggesting the depicted event preference observed in previous studies generalizes to visual gender cues. Stereotypicality match did not seem to modulate this gaze behavior. However, when there was a mismatch between the sentence and the previous video, participants tended to look away from the target face (post-verbally for action-verb mismatches and at the final subject region for hand gender -subject gender mismatches), suggesting outright picture-sentence incongruence can modulate the preference to inspect the face whose gender matched that of the hands seen in the previous video

Visual gender cues elicit agent expectations: different mismatches in situated language comprehension

Rodriguez A, Burigo M, Knoeferle P. Visual gender cues elicit agent expectations: different mismatches in situated language comprehension. In: Airenti G, Bara BG, Sandini G, eds. Proceedings of the EuroAsianPacific Joint Conference on Cognitive Science (EAPCogSci 2015). CEUR Workshop Proceedings. Vol 1419. Aachen; 2015: 234-239

Eye-tracking situated language comprehension: Immediate actor gaze versus recent action events

Abashidze D, Knoeferle P, Carminati MN. Eye-tracking situated language comprehension: Immediate actor gaze versus recent action events. In: Proceedings of the 37th Annual Meeting of the Cognitive Science Society. Pasadena, California, USA; 2015

How robust is the recent-event preference?

Abashidze D, Carminati MN, Knoeferle P. How robust is the recent-event preference? In: Proceedings of the 36th Annual Meeting of the Cognitive Science Society. Austin, TX: Cognitive Science Society; 2014: 92-97