Eye gaze metrics reflect a shared motor representation for action observation and movement imagery

Action observation (AO) and movement imagery (MI) have been reported to share similar neural networks. This study investigated the congruency between AO and MI using the eye gaze metrics, dwell time and fixation number. A simple reach-grasp-place arm movement was observed and, in a second condition, imagined where the movement was presented from the first person perspective (1PP) and the third person perspective (3PP). Dwell time and number of fixations were calculated for whole scene and regions of interest (ROIs). For whole scene, no significant differences were found in the number of fixations for condition (AO, MI) or perspective. Dwell time, however, was significantly longer in AO than MI. For ROIs, the number of fixations was significantly greater in 1PP than 3PP. The data provide support for congruence between motor simulation states but also indicate some functional differences

Overt orienting of spatial attention and corticospinal excitability during action observation are unrelated

Observing moving body parts can automatically activate topographically corresponding motor representations in the primary motor cortex (M1), the so-called direct matching. Novel neurophysiological findings from social contexts are nonetheless proving that this process is not automatic as previously thought. The motor system can flexibly shift from imitative to incongruent motor preparation, when requested by a social gesture. In the present study we aim to bring an increase in the literature by assessing whether and how diverting overt spatial attention might affect motor preparation in contexts requiring interactive responses from the onlooker. Experiment 1 shows that overt attention-although anchored to an observed biological movement-can be captured by a target object as soon as a social request for it becomes evident. Experiment 2 reveals that the appearance of a short-lasting red dot in the contralateral space can divert attention from the target, but not from the biological movement. Nevertheless, transcranial magnetic stimulation (TMS) over M1 combined with electromyography (EMG) recordings (Experiment 3) indicates that attentional interference reduces corticospinal excitability related to the observed movement, but not motor preparation for a complementary action on the target. This work provides evidence that social motor preparation is impermeable to attentional interference and that a double dissociation is present between overt orienting of spatial attention and neurophysiological markers of action observation

Effects of Social and Non-Social Interpretations of Complex Images on Human Eye Movement and Brain Activation

Communicating and interacting with others is an essential part of our daily routines as humans. Performing these actions appropriately requires the ability to identify, extract, and process salient social cues from the environment. The subsequent application of such knowledge is important for inferring and predicting the behavior of other people. The eyes and brain must work together to fixate and process only the most critical social signals within a scene while passing over and / or completely ignoring other aspects of the scene. While brain activation to isolated presentations of objects and people presentations have been characterized, information about the brain\u27s activation patterns to more comprehensive scenes containing multiple categories of information is limited. Furthermore, little is known about how different interpretations of a scene might alter how that scene is viewed or how the brain responds to that scene. Therefore, the studies presented herein used a combination of infrared eye tracking and functional magnetic resonance imaging techniques to investigate the eye movement and brain activation patterns to socially- and non-socially-relevant interpretations of the same set of complex stimuli. Eye tracking data showed that each gaze pattern was consistent with viewing and attending to only one category of information (people or objects) despite both categories being present in all images. Functional magnetic resonance imaging revealed that a region of the right superior temporal sulcus was selectively activated by the social condition compared to the non-social condition, an area known for its role in social tasks. Brain activation in response to the non-social condition was located in many of the same regions associated with the recognition and processing of visual objects presented in isolation. Taken together, these results demonstrate that in healthy adults, eye movement and brain activation patterns to identical scenes change markedly as a function of attentional focus and interpretation intention. Utilizing realistic and complex stimuli to study the eye gaze and neural activation patterns associated with processing social versus non-social information in the healthy brain is an important step towards understanding the deficits present in individuals with social cognition disorders like autism and schizophrenia

The development of prospective processing of simple and complex actions in early childhood

Everyday human life is characterized by social interactions, for which understanding and anticipating others’ actions play a crucial role. Furthermore, planning one’s movements in an anticipatory manner allows for acting smoothly and efficiently by avoiding time-consuming corrective movements. This thesis aimed to investigate how prospective processing of others’ actions as well as of children’s own actions develops over early childhood. One key focus was to examine the prospective processing of actions of varying degrees of complexity. Four empirical studies were conducted to expand our understanding of this topic by focusing on the following three aspects. First, Study 1 examined two possible mechanisms underlying children’s visual anticipations (as a key measure of prospective action understanding) during perceiving others’ simple actions. It aimed at answering the question of whether simulation theories or lower-level perceptual mechanisms account for children’s visual anticipations of others’ simple grasping actions. Three-year-olds’, 4-year-olds’, 10-year olds’, and adults’ (N = 98) prospective action processing was assessed by their anticipatory gaze shifts within an eye-tracking study. Participants observed a hand reaching for one out of two objects while the motor kinematics of the reaching hand varied depending on whether it reached for a close or far object. Results revealed that none of the children age groups used the motor kinematics to correctly visually anticipate the target object. Instead, they showed a looking bias to the close object. This indicates that lower-level perceptual mechanisms (following the general movement of the reaching hand and being attracted by a salient object) and not motor simulation (matching the observed action onto their own motor repertoire) accounted for children’s visual anticipations of simple other’s simple grasping movements. Second, the roles of automatic and controlled processes for children’s prospective processing of others’ actions as well as for their prospective action planning were examined. The results of Study 1 mentioned above indicated that children’s visual anticipations were triggered rather automatically. The second experiment of Study 1 assessed participants’ (N = 80) verbal predictions of the grasping actions described above. It turned out that 10-year-olds used kinematic cues to verbally predict the action target and that 4-year-olds learned to do so over the trials, whereas neither of the age groups correctly visually anticipated the action in the first experiment. These findings suggest that a more sophisticated understanding of other’s actions might initially require explicit, controlled processing of the situation. This conflicts with the predominant view in developmental psychology that children show an implicit understanding of others’ actions before showing an explicit understanding later in ontogeny. Furthermore, Study 2 examined the role of automatic and controlled processes for children’s prospective action planning in a large sample (N = 246) of 2- to 14-year-old children and adults. Two conceptually different object manipulation tasks were assessed to investigate whether anticipatory movement planning develops as general capacity and to what extent participants rely on the habitual use of the object or plan their grasping movement in a controlled manner. The results suggest that anticipatory movement planning might develop as a general capacity. Furthermore, the findings are indicative of an increase of controlled processes for prospective action planning in early childhood, while the developmental pattern of the interplay of automatic and controlled processes was task-specific. In sum, Study 2 highlights the developmental dynamics of the interplay of habitual and controlled processes in children’s own prospective action planning. Third, Study 3a and 3b examined in more detail how the ability to prospectively process hierarchical actions develops in early childhood and whether it is related to children’s processing of hierarchical structures in the language domain. In Study 3a, 3- to 6-year olds and adults (N = 111) were asked to verbally predict actions that required a means (sub-action) to achieve an actor’s stated overarching goal. This task also served as one out of three complex action tasks in Study 3b, which investigated 3- to 6-year-old children’s (N = 130) hierarchical structure processing across the action and language domain. Overall, the results suggest that children’s ability to predict hierarchical actions increases over early childhood (Study 3a, 3b). Furthermore, Study 3b indicates that the processing of hierarchical structures might develop as a domain-general capacity and highlights the role of working memory for hierarchical structure processing. Taken together, this thesis contributes to research on how young children prospectively understand others’ actions and how they prospectively plan their own actions. First, it provides evidence that low-level perceptual mechanisms rather than motor simulation account for young children’s visual anticipations of simple grasping actions (Study 1). Second, it points to substantial developments of controlled prospective processing over early childhood, which is supposed to be essential for processing more complex situations and actions (Study 1, 2, 3a, 3b). Lastly, children’s development of prospectively processing hierarchical action and language structures seems related and might be explained by working memory (Study 3b).Unser tägliches Leben ist von sozialen Interaktionen geprägt. Die Handlungen anderer zu verstehen und antizipieren zu können ist für soziale Interaktionen von großer Bedeutung. Darüber hinaus ermöglicht eine antizipative Bewegungsplanung nahtlos ineinander übergehende und effiziente Handlungen ohne zeitaufwendige Korrekturbewegungen. Das Ziel dieser Dissertation ist zu untersuchen wie sich die prospektive Verarbeitung von Handlungen anderer Personen, als auch die eigene prospektive Handlungsplanung in der frühen Kindheit entwickelt. Ein zentraler Punkt der vorliegenden Arbeit ist, dass sie die prospektive Verarbeitung von Handlungen unterschiedlicher Komplexität betrachtet. Um unser Verständnis dieses Forschungsbereichs zu erweitern, wurden vier empirische Studien mit einem Fokus auf die folgenden drei Teilaspekte durchgeführt. Erstens untersuchte Studie 1 zwei mögliche zugrunde liegende Mechanismen für visuelle Antizipationen (als ein zentrales Maß für prospektives Handlungsverständnis) von Kindern, während die Kinder eine einfache Handlung einer anderen Person beobachteten. Die Studie hatte zum Ziel herauszufinden, ob visuelle Antizipationen einfacher Greifbewegungen anderer durch Simulationstheorien oder durch Low-Level Wahrnehmungsmechanismen erklärt werden können. Die prospektive Handlungsverarbeitung 3-Jähriger, 4-Jähriger, 10-Jähriger und Erwachsener (N = 98) wurde anhand ihrer antizipatorischen Blickbewegungen im Rahmen einer Eye-Tracking Studie erfasst. Die Studienteilnehmer beobachteten, wie eine Hand nach einem von zwei Objekten griff. Die motorische Kinematik der greifenden Hand variierte in Abhängigkeit davon, ob ein näheres oder weiter entferntes Objekt gegriffen wurde. Die Ergebnisse zeigten, dass keine der Kinder Altersgruppen die kinematischen Informationen nutzte, um das korrekte Zielobjekt zu antizipieren. Stattdessen antizipierten sie tendenziell das nähere Objekt, unabhängig von der motorischen Information. Dies deutet darauf hin, dass nicht die Simulation von Bewegungen (spiegeln einer beobachteten Handlung auf das eigene motorische Repertoire), sondern Low-Level Wahrnehmungsmechanismen (Antizipieren der Bewegungsrichtung der Hand und gleichzeitiges Angezogen werden von einem salienten Objekt) die visuellen Antizipationen der Kinder erklärten. Zweitens wurde die Rolle automatischer und kontrollierter Prozesse für die prospektive Handlungsverarbeitung von Handlungen anderer Personen, als auch für die eigene prospektive Handlungsplanung in der frühen Kindheit untersucht. Die eben erwähnten Ergebnisse der ersten Studie wiesen darauf hin, dass die visuellen Antizipationen eher automatisch hervorgerufen wurden. In einem zweiten Experiment von Studie 1 sollten die Versuchspersonen (N = 80) die beschriebenen Greifbewegungen verbal vorhersagen. Es zeigte sich, dass 10-Jährige die kinematischen Informationen nutzten, um vorherzusagen welches der beiden Objekte gegriffen wird und dass 4-Jährige über die Trials hinweg lernten eine korrekte Vorhersage zu treffen, wohingegen keine der beiden Altersgruppen die Handlung im ersten Experiment korrekt antizipierte. Diese Befunde deuten darauf hin, dass ein differenzierteres Verständnis von Handlungen anderer Personen zunächst eine explizite, kontrollierte Verarbeitung der Situation erfordern kann. Dies steht im Konflikt mit der vorherrschenden Sichtweise in der Entwicklungspsychologie, dass Kinder ein implizites Verständnis von Handlungen anderer Personen haben, das sich erst später in der Ontogenese in ihrem expliziten Verständnis widerspiegelt. Darüber hinaus erforschte Studie 2 die Rolle automatischer und kontrollierter Prozesse für die eigene prospektive Handlungsplanung der Kinder in einer großen Stichprobe (N = 246) von 2- bis 14-jährigen Kindern und Erwachsenen. Zwei konzeptionell unterschiedliche Objektmanipulationsaufgaben wurden erhoben. Einerseits, um herauszufinden, ob sich die antizipatorische Bewegungsplanung als eine allgemeine Fähigkeit entwickelt. Andererseits, um zu untersuchen, inwiefern das Greifen der Objekte von deren gewöhnlichen Gebrauch (automatisch) geleitet oder von kontrollierten Prozessen gesteuert wird. Die Ergebnisse deuten darauf hin, dass sich die antizipatorische Bewegungsplanung als allgemeine Fähigkeit entwickelt. Darüber hinaus lassen sie auf eine Zunahme von kontrollierten Prozessen für die prospektive Handlungsplanung in der frühen Kindheit schließen, während sich das Entwicklungsmuster des Zusammenspiels von automatischen und kontrollierten Prozessen als aufgabenspezifisch herausstellte. Zusammengefasst hebt Studie 2 die Dynamik der Entwicklung des Zusammenspiels von automatischen und kontrollierten Prozessen für die prospektive Handlungsplanung hervor. Drittens wurde in den Studien 3a und 3b genauer untersucht, wie sich die Fähigkeit hierarchische Handlungen prospektiv zu verarbeiten in der frühen Kindheit entwickelt und ob sie mit der Entwicklung der Verarbeitung von hierarchische Sprachstrukturen zusammenhängt. In Studie 3a sollten 3- bis 6-jährige Kinder und Erwachsene (N = 111) Handlungen vorhersagen, die ein Mittel (eine Unterhandlung) erforderten, um ein angegebenes übergeordnetes Ziel eines Akteurs erreichen zu können. Diese Aufgabe diente auch als eine von drei hierarchischen Handlungsaufgaben in Studie 3b, in der die Verarbeitung hierarchischer Strukturen in 3- bis 6-jährigen Kindern (N = 130) in der Handlungs- und Sprachdomäne verglichen wurde. Zusammengefasst sprechen die Ergebnisse der Studien 3a und 3b dafür, dass die Fähigkeit hierarchische Handlungen vorherzusagen in der frühen Kindheit zunimmt. Ferner deuten die Ergebnisse darauf hin, dass sich die Verarbeitung hierarchischer Strukturen als domänenübergreifende Fähigkeit entwickelt und betonen die Rolle des Arbeitsgedächtnisses für die Verarbeitung hierarchischer Strukturen (Studie 3b). Zusammengefasst trägt diese Dissertation zur Erforschung des prospektiven Handlungsverständnisses sowie der prospektiven Handlungsplanung in der frühen Kindheit bei. Erstens deuten die Ergebnisse darauf hin, dass Low-Level Wahrnehmungsmechanismen visuelle Antizipationen einfacher Greifhandlungen in der Kindheit erklären, während keine Evidenz für Simulationstheorien gefunden wurde (Studie 1). Zweitens weisen die Befunde auf eine erhebliche Entwicklung kontrollierter, prospektiver Verarbeitung von Handlungen in der frühen Kindheit hin, welche als essentiell angesehen wird, um komplexere Situationen und Handlungen zu verarbeiten (Studien 1, 2, 3a, 3b). Zuletzt scheint es einen Zusammenhang zwischen der Entwicklung der Verarbeitung von komplexen Handlungs- und Sprachstrukturen in der frühen Kindheit zu geben, welcher vermutlich durch das Arbeitsgedächtnis erklärt werden kann (Studie 3b)

Toddlers' action prediction: statistical learning of continuous action sequences

The current eye-tracking study investigated whether toddlers use statistical information to make anticipatory eye movements while observing continuous action sequences. In two conditions, 19-month-old participants watched either a person performing an action sequence (Agent condition) or a self-propelled visual event sequence (Ghost condition). Both sequences featured a statistical structure in which certain action pairs occurred with deterministic transitional probabilities. Toddlers learned the transitional probabilities between the action steps of the deterministic action pairs and made predictive fixations to the location of the next action in the Agent condition but not in the Ghost condition. These findings suggest that young toddlers gain unique information from the statistical structure contained within action sequences and are able to successfully predict upcoming action steps based on this acquired knowledge. Furthermore, predictive gaze behavior was correlated with reproduction of sequential actions following exposure to statistical regularities. This study extends previous developmental work by showing that statistical learning can guide the emergence of anticipatory eye movements during observation of continuous action sequences

Looking ahead: anticipatory gaze and motor ability in infancy

The present study asks when infants are able to selectively anticipate the goals of observed actions, and how this ability relates to infants' own abilities to produce those specific actions. Using eye-tracking technology to measure on-line anticipation, 6-, 8- and 10-month-old infants and a control group of adults were tested while observing an adult reach with a whole hand grasp, a precision grasp or a closed fist towards one of two different sized objects. The same infants were also given a comparable action production task. All infants showed proactive gaze to the whole hand grasps, with increased degrees of proactivity in the older groups. Gaze proactivity to the precision grasps, however, was present from 8 months of age. Moreover, the infants' ability in performing precision grasping strongly predicted their ability in using the actor's hand shape cues to differentially anticipate the goal of the observed action, even when age was partialled out. The results are discussed in terms of the specificity of action anticipation, and the fine-grained relationship between action production and action perception

Humanoid-based protocols to study social cognition

Social cognition is broadly defined as the way humans understand and process their interactions with other humans. In recent years, humans have become more and more used to interact with non-human agents, such as technological artifacts. Although these interactions have been restricted to human-controlled artifacts, they will soon include interactions with embodied and autonomous mechanical agents, i.e., robots. This challenge has motivated an area of research related to the investigation of human reactions towards robots, widely referred to as Human-Robot Interaction (HRI). Classical HRI protocols often rely on explicit measures, e.g., subjective reports. Therefore, they cannot address the quantification of the crucial implicit social cognitive processes that are evoked during an interaction. This thesis aims to develop a link between cognitive neuroscience and human-robot interaction (HRI) to study social cognition. This approach overcomes methodological constraints of both fields, allowing to trigger and capture the mechanisms of real-life social interactions while ensuring high experimental control. The present PhD work demonstrates this through the systematic study of the effect of online eye contact on gaze-mediated orienting of attention. The study presented in Publication I aims to adapt the gaze-cueing paradigm from cognitive science to an objective neuroscientific HRI protocol. Furthermore, it investigates whether the gaze-mediated orienting of attention is sensitive to the establishment of eye contact. The study replicates classic screen-based findings of attentional orienting mediated by gaze both at behavioral and neural levels, highlighting the feasibility and the scientific value of adding neuroscientific methods to HRI protocols. The aim of the study presented in Publication II is to examine whether and how real-time eye contact affects the dual-component model of joint attention orienting. To this end, cue validity and stimulus-to-onset asynchrony are also manipulated. The results show an interactive effect of strategic (cue validity) and social (eye contact) top-down components on the botton-up reflexive component of gaze-mediated orienting of attention. The study presented in Publication III aims to examine the subjective engagement and attribution of human likeness towards the robot depending on established eye contact or not during a joint attention task. Subjective reports show that eye contact increases human likeness attribution and feelings of engagement with the robot compared to a no-eye contact condition. The aim of the study presented in Publication IV is to investigate whether eye contact established by a humanoid robot affects objective measures of engagement (i.e. joint attention and fixation durations), and subjective feelings of engagement with the robot during a joint attention task. Results show that eye contact modulates attentional engagement, with longer fixations at the robot’s face and cueing effect when the robot establishes eye contact. In contrast, subjective reports show that the feeling of being engaged with the robot in an HRI protocol is not modulated by real-time eye contact. This study further supports the necessity for adding objective methods to HRI. Overall, this PhD work shows that embodied artificial agents can advance the theoretical knowledge of social cognitive mechanisms by serving as sophisticated interactive stimuli of high ecological validity and excellent experimental control. Moreover, humanoid-based protocols grounded in cognitive science can advance the HRI community by informing about the exact cognitive mechanisms that are present during HRI