Humanoid-based protocols to study social cognition

Social cognition is broadly defined as the way humans understand and process their interactions with other humans. In recent years, humans have become more and more used to interact with non-human agents, such as technological artifacts. Although these interactions have been restricted to human-controlled artifacts, they will soon include interactions with embodied and autonomous mechanical agents, i.e., robots. This challenge has motivated an area of research related to the investigation of human reactions towards robots, widely referred to as Human-Robot Interaction (HRI). Classical HRI protocols often rely on explicit measures, e.g., subjective reports. Therefore, they cannot address the quantification of the crucial implicit social cognitive processes that are evoked during an interaction. This thesis aims to develop a link between cognitive neuroscience and human-robot interaction (HRI) to study social cognition. This approach overcomes methodological constraints of both fields, allowing to trigger and capture the mechanisms of real-life social interactions while ensuring high experimental control. The present PhD work demonstrates this through the systematic study of the effect of online eye contact on gaze-mediated orienting of attention. The study presented in Publication I aims to adapt the gaze-cueing paradigm from cognitive science to an objective neuroscientific HRI protocol. Furthermore, it investigates whether the gaze-mediated orienting of attention is sensitive to the establishment of eye contact. The study replicates classic screen-based findings of attentional orienting mediated by gaze both at behavioral and neural levels, highlighting the feasibility and the scientific value of adding neuroscientific methods to HRI protocols. The aim of the study presented in Publication II is to examine whether and how real-time eye contact affects the dual-component model of joint attention orienting. To this end, cue validity and stimulus-to-onset asynchrony are also manipulated. The results show an interactive effect of strategic (cue validity) and social (eye contact) top-down components on the botton-up reflexive component of gaze-mediated orienting of attention. The study presented in Publication III aims to examine the subjective engagement and attribution of human likeness towards the robot depending on established eye contact or not during a joint attention task. Subjective reports show that eye contact increases human likeness attribution and feelings of engagement with the robot compared to a no-eye contact condition. The aim of the study presented in Publication IV is to investigate whether eye contact established by a humanoid robot affects objective measures of engagement (i.e. joint attention and fixation durations), and subjective feelings of engagement with the robot during a joint attention task. Results show that eye contact modulates attentional engagement, with longer fixations at the robot’s face and cueing effect when the robot establishes eye contact. In contrast, subjective reports show that the feeling of being engaged with the robot in an HRI protocol is not modulated by real-time eye contact. This study further supports the necessity for adding objective methods to HRI. Overall, this PhD work shows that embodied artificial agents can advance the theoretical knowledge of social cognitive mechanisms by serving as sophisticated interactive stimuli of high ecological validity and excellent experimental control. Moreover, humanoid-based protocols grounded in cognitive science can advance the HRI community by informing about the exact cognitive mechanisms that are present during HRI

On the role of eye contact in gaze cueing

Most experimental protocols examining joint attention with the gaze cueing paradigm are "observational" and "offline", thereby not involving social interaction. We examined whether within a naturalistic online interaction, real-time eye contact influences the gaze cueing effect (GCE). We embedded gaze cueing in an interactive protocol with the iCub humanoid robot. This has the advantage of ecological validity combined with excellent experimental control. Critically, before averting the gaze, iCub either established eye contact or not, a manipulation enabled by an algorithm detecting position of the human eyes. For non-predictive gaze cueing procedure (Experiment 1), only the eye contact condition elicited GCE, while for counter-predictive procedure (Experiment 2), only the condition with no eye contact induced GCE. These results reveal an interactive effect of strategic (gaze validity) and social (eye contact) top-down components on the reflexive orienting of attention induced by gaze cues. More generally, we propose that naturalistic protocols with an embodied presence of an agent can cast a new light on mechanisms of social cognition

Eye contact during joint attention with a humanoid robot modulates oscillatory brain activity

Eye contact established by a human partner has been shown to affect various cognitive processes of the receiver. However, little is known about humans’ responses to eye contact established by a humanoid robot. Here, we aimed at examining humans’ oscillatory brain response to eye contact with a humanoid robot. Eye contact (or lack thereof) was embedded in a gaze cueing task and preceded the phase of gaze-related attentional orienting. In addition to examining the effect of eye contact on the recipient, we also tested its impact on gaze cueing effects. Results showed that participants rated eye contact as more engaging and responded with higher desynchronization of alpha-band activity in left fronto-central and central electrode clusters when the robot established eye contact with them, compared to no eye contact condition. However, eye contact did not modulate gaze cueing effects. The results are interpreted in terms of the functional roles involved in alpha central rhythms (potentially interpretable also as mu rhythm), including joint attention and engagement in social interaction

To follow or not to follow your gaze: “The interplay between strategic control and the eye contact effect on gaze-induced attentional orienting

The current study examines how the social component of eye contact interacts with the top-down strategic control on the modulation of gaze-mediated orienting of attention

Embodiment matters when establishing eye contact with a robot

Eye contact constitutes a strong social signal in humans and affects various attentional processes. However, eye contact with another human might evoke different responses compared with a direct gaze depicted on a screen. Previous embodied gaze-cueing experiments with iCub humanoid robot showed eye contact modulations on gaze-cueing effect using a long stimulus-onset-asynchrony (SOA: 1000 ms, no predictive gaze cue). Instead, eye contact did not modulate the gaze-cueing effect using a shorter SOA (500 ms, no predictive gaze cue). In the present study, we investigated whether a robot’s eye contact on the screen could modulate the gaze-cueing effect by adapting the previous embodied experiments to a screen-based setup. Specifically, in two experiments we examined the impact of eye contact on the gaze-cueing effect for non-predictive cues while we varied the SOA (500 ms and 1000 ms). Our results showed that the robot’s direct gaze did not modulate the gaze-cueing effect (gaze-cueing effect present in all conditions), thereby suggesting that direct gaze presented in a 2D format on the screen has less impact on observers than its 3D embodied version in a physically present robot. Overall, our findings stress the importance of embodied interactions for understanding the mechanisms of social cognition

The fine-grained temporal dynamics of social timing: a window into sociality of embodied social agents. Comment on “The evolution of social timing” by L. Verga, S. A. Kotz, & A. Ravignani

Verga, Kotz, and Ravignani’s paper on “The evolution of social timing” [1] provides an elegant proposal into how sociality and timing are interconnected and ubiquitous in human and nonhuman animal interactions, and proposes a framework to investigate the evolution of social timing and why it is so fundamental to social animals. The authors outline some of the key mechanisms that support social timing, such as interpersonal synchronization and mutual adaptation, but do not go into depth with respect to the fine-grained temporal properties and mechanisms underlying individual and interpersonal signals, which may lead to different social outcomes. Here we propose that these fine-grained temporal mechanisms are key to understanding the social timing and consequently sociality of embodied social living species. We first summarize the fine-grained temporal mechanisms in human interactions that we argue are at the very core of social interactions, and show how these can predict important social consequences. Next, we examine whether such mechanisms can lead to varying perceptions of sociality, attribution of agency, and interaction parameters when embedded in artificial embodied agents (e.g., social robots). Finally, we argue that the fine-grained individual and interpersonal temporal dynamics are the key social mechanisms lacking in disembodied artificial intelligence (AI), such as ChatGPT, which ultimately reveals its lack of sociality