Eyes on the mind : investigating the influence of gaze dynamics on the perception of others in real-time social interaction

ACKNOWLEDGMENTS This study was partially supported by a grant of the Köln Fortune Program of the Medical Faculty at the University of Cologne to Leonhard Schilbach and by a grant “Other Minds” of the German Ministry of Research and Education to Kai Vogeley. The authors would like to thank Stephanie Alexius and Leonhard Engels for their assistance in data collection.Peer reviewedPublisher PD

The Ascription of Intentions in Gaze-Contingent Social Encounters

Observing the gaze of others not only helps us understand their motives and intentions but also to engage in interactions with them. Therefore, one of the most important assessments we have to do when observing someone’s gaze is whether their gaze behavior is meant as a signal to us. For persons with difficulties in understanding and interpreting the intentions of others, as is the case in autism spectrum disorder (ASD), this situation can be especially challenging. This thesis addressed the problem of how we are able to interpret and understand the gaze behavior of others in order to communicate successfully. As a first step, it examines which kinds of inferences regarding the general intentions of another person are possible from the passive observation of gaze behavior (study 1). Turning towards gaze interactions, it then outlined a theoretical concept and a taxonomy of social gaze, allowing for holistic considerations of ongoing gaze encounters (study 2). The practical implementation of this approach in form of the new agent interaction platform TriPy was presented in study 3. Subsequently, an investigation of the inference of communicative intentions in ongoing gaze interactions was performed (study 4). Study 5 then compared the performance of healthy participants with that of persons with ASD, finding that the latter especially have trouble in interactive situations. As an outlook to the future direction of social gaze investigations and their application in clinical contexts, study 6 introduced a new technical system for avatar-mediated communication between two persons combined with machine learning based data analysis. Before these studies are presented in more detail, a theoretical background is provided by introducing central phenomena of social gaze as well as describing impairments in gaze communication in ASD. In addition, methodological requirements and challenges in the investigation of social gaze are elucidated. After presenting the individual studies, the results obtained in this thesis are integrated in a general discussion focusing on our understanding of social gaze, its clinical implications, as well as potential future directions in social gaze research

The relation between gaze and turn-taking in dyadic avatar-mediated conversations

Human communication comprises a complex and dynamic interplay of verbal and nonverbal communication channels. The investigation of these channels therefore represents a major methodological challenge. Technical developments in interaction platforms using virtual characters provide tools for these investigations. Paradigms in which participants interact with algorithmically controlled agents have already enabled the investigation of individual nonverbal communication channels with the necessary experimental control. However, it is unclear how these results relate to human-human communication. Here, we present a study with a new system for human-human interactions mediated by avatars. As a proof-of-concept, we tested the generalisability of gaze patterns during turn-taking in avatar-mediated conversations. Results show that given our Bayesian mixed effects model, priors and data, there is compelling evidence that gaze patterns are comparable to natural interactions. Exploratory analyses show that our system is suitable to shed light on variability in individual-specific gaze behaviour, which we plan to investigate further

The Temporality of Situated Cognition

Situated cognition embeds perceptions, thoughts, and behavior within the contextual framework of so-called 4E cognition understanding cognition to be embodied, enactive, extended, and embedded. Whereas this definition is primarily based on the spatial properties of a situation, it neglects a fundamental constituent: the cognitive situation asenduring. On a subpersonal level, situated cognition requires the integration of information processing within a minimal temporal extension generating the basic building blocks of perception and action (microlayer of time). On a personal level, lived situations and experienced narratives leading to our biography can be defined by their broader temporal horizons (macrolayer of time). The macrolayer of time is based on and emerges from information processing on the microlayer of time. Whereas the constraints on the microlayer are primarily defined by the integrity of neurobiological processes within an individual cognitive system, the temporal horizons and subsequently the situational context on the macrolayer are defined by the complex affordances of a situation on a personal or interpersonal level. On both time layers, cognition can be defined as a continuous dynamic process, reflecting the transition from one situated state to another. Taken together, the events forming the delimiting horizons of these situations correspond to the temporal structure of the cognitive process along which it continuously proceeds. The dynamic driving and enabling this transition from state to state is synonymous with the inherent flow of time. Just as the layers of time, flow and structure, are inseparably connected. The integration of temporal flow and temporal structure into the continuous dynamic process constitutes the enduring situatedness of cognition. By providing everyday examples and examples from psychopathology, we highlight the benefits of understanding cognitive processes as part of enduring situations

Temporal Behavioral Parameters of On-Going Gaze Encounters in a Virtual Environment

To navigate the social world, humans heavily rely on gaze for non-verbal communication as it conveys information in a highly dynamic and complex, yet concise manner: For instance, humans utilize gaze effortlessly to direct and infer the attention of a possible interaction partner. Many traditional paradigms in social gaze research though rely on static ways of assessing gaze interaction, e.g., by using images or prerecorded videos as stimulus material. Emerging gaze contingent paradigms, in which algorithmically controlled virtual characters can respond flexibly to the gaze behavior of humans, provide high ecological validity. Ideally, these are based on models of human behavior which allow for precise, parameterized characterization of behavior, and should include variable interactive settings and different communicative states of the interacting agents. The present study provides a complete definition and empirical description of a behavioral parameter space of human gaze behavior in extended gaze encounters. To this end, we (i) modeled a shared 2D virtual environment on a computer screen in which a human could interact via gaze with an agent and simultaneously presented objects to create instances of joint attention and (ii) determined quantitatively the free model parameters (temporal and probabilistic) of behavior within this environment to provide a first complete, detailed description of the behavioral parameter space governing joint attention. This knowledge is essential to enable the modeling of interacting agents with a high degree of ecological validity, be it for cognitive studies or applications in human-robot interaction

The “Social Gaze Space”: A Taxonomy for Gaze-Based Communication in Triadic Interactions

Humans substantially rely on non-verbal cues in their communication and interaction with others. The eyes represent a “simultaneous input-output device”: While we observe others and obtain information about their mental states (including feelings, thoughts, and intentions-to-act), our gaze simultaneously provides information about our own attention and inner experiences. This substantiates its pivotal role for the coordination of communication. The communicative and coordinative capacities – and their phylogenetic and ontogenetic impacts – become fully apparent in triadic interactions constituted in its simplest form by two persons and an object. Technological advances have sparked renewed interest in social gaze and provide new methodological approaches. Here we introduce the ‘Social Gaze Space’ as a new conceptual framework for the systematic study of gaze behavior during social information processing. It covers all possible categorical states, namely ‘partner-oriented,’ ‘object-oriented,’ ‘introspective,’ ‘initiating joint attention,’ and ‘responding joint attention.’ Different combinations of these states explain several interpersonal phenomena. We argue that this taxonomy distinguishes the most relevant interactional states along their distinctive features, and will showcase the implications for prominent social gaze phenomena. The taxonomy allows to identify research desiderates that have been neglected so far. We argue for a systematic investigation of these phenomena and discuss some related methodological issues

Inferring Interactivity From Gaze Patterns During Triadic Person-Object-Agent Interactions

Observing others' gaze informs us about relevant matters in the environment. Humans' sensitivity to gaze cues and our ability to use this information to focus our own attention is crucial to learning, social coordination, and survival. Gaze can also be a deliberate social signal which captures and directs the gaze of others toward an object of interest. In the current study, we investigated whether the intention to actively communicate one's own attentional focus can be inferred from the dynamics of gaze alone. We used a triadic gaze interaction paradigm based on the recently proposed classification of attentional states and respective gaze patterns in person-object-person interactions, the so-called social gaze space (SGS). Twenty-eight participants interacted with a computer controlled virtual agent while they assumed to interact with a real human. During the experiment, the virtual agent engaged in various gaze patterns which were determined by the agent's attentional communicative state, as described by the concept of SGS. After each interaction, participants were asked to judge whether the other person was trying to deliberately interact with them. Results show that participants were able to infer the communicative intention solely from the agent's gaze behavior. The results substantiate claims about the pivotal role of gaze in social coordination and relationship formation. Our results further reveal that social expectations are reflected in differential responses to the displayed gaze patterns and may be crucial for impression formation during gaze-based interaction. To the best of our knowledge, this is the first study to document the experience of interactivity in continuous and contingent triadic gaze interactions

Distinguishing Social From Private Intentions Through the Passive Observation of Gaze Cues

Observing others’ gaze is most informative during social encounters between humans: We can learn about potentially salient objects in the shared environment, infer others’ mental states and detect their communicative intentions. We almost automatically follow the gaze of others in order to check the relevance of the target of the other’s attention. This phenomenon called gaze cueing can be conceptualized as a triadic interaction involving a gaze initiator, a gaze follower and a gaze target, i.e., an object or person of interest in the environment. Gaze cueing can occur as “gaze pointing” with a communicative or “social” intention by the initiator, telling the observer that she/he is meant to follow, or as an incidental event, in which the observer follows spontaneously without any intention of the observed person. Here, we investigate which gaze cues let an observer ascribe a social intention to the observed person’s gaze and whether and to which degree previous eye contact in combination with an object fixation contributes to this ascription. We varied the orientation of the starting position of gaze toward the observer and the orientation of the end position of a lateral gaze shift. In two experiments participants had to infer from the gaze behavior either mere approach (“the person looked at me”) vs. a social (“the person wanted to show me something”) or a social vs. a private motivation (“the person was interested in something”). Participants differentially attributed either approach behavior, a social, or a private intention to the agent solely based on the passive observation of the two specific gaze cues of start and end position. While for the attribution of privately motivated behavior, participants relied solely on the end position of the gaze shift, the social interpretation of the observed behavior depended additionally upon initial eye contact. Implications of these results for future social gaze and social cognition research in general are discussed

Dissociating passage and duration of time experiences through the intensity of ongoing visual change

The experience of passage of time is assumed to be a constitutive component of our subjective phenomenal experience and our everyday life that is detached from the estimation of time durations. However, our understanding of the factors contributing to passage of time experience has been mostly restricted to associated emotional and cognitive experiences in temporally extended situations. Here, we tested the influence of low-level visual stimuli on the experience of passage and duration of time in 10–30 s intervals. We introduce a new paradigm in a starfield environment that allows to study the effects of basic visual aspects of a scene (velocity and density of stars in the starfield) and the duration of the situation, both embedded in a color tracking task. Results from two experiments show that velocity and density of stars in the starfield affect passage of time experience independent from duration estimation and the color tracking task: the experienced passage of time is accelerated with higher rates of moment-to-moment changes in the starfield while duration estimations are comparably unaffected. The results strongly suggest differential psychological processes underlying the experience of time passing by and the ability to estimate time durations. Potential mechanisms behind these results and the prospects of experimental approaches towards passage of time experience in psychological and neuroscientific research are discussed

The “Social Gaze Space” : A Taxonomy for Gaze-Based Communication in Triadic Interactions

Humans substantially rely on non-verbal cues in their communication and interaction with others. The eyes represent a “simultaneous input-output device”: While we observe others and obtain information about their mental states (including feelings, thoughts, and intentions-to-act), our gaze simultaneously provides information about our own attention and inner experiences. This substantiates its pivotal role for the coordination of communication. The communicative and coordinative capacities – and their phylogenetic and ontogenetic impacts – become fully apparent in triadic interactions constituted in its simplest form by two persons and an object. Technological advances have sparked renewed interest in social gaze and provide new methodological approaches. Here we introduce the ‘Social Gaze Space’ as a new conceptual framework for the systematic study of gaze behavior during social information processing. It covers all possible categorical states, namely ‘partner-oriented,’ ‘object-oriented,’ ‘introspective,’ ‘initiating joint attention,’ and ‘responding joint attention.’ Different combinations of these states explain several interpersonal phenomena. We argue that this taxonomy distinguishes the most relevant interactional states along their distinctive features, and will showcase the implications for prominent social gaze phenomena. The taxonomy allows to identify research desiderates that have been neglected so far. We argue for a systematic investigation of these phenomena and discuss some related methodological issues