Brain interaction during cooperation: Evaluating local properties of multiple-brain network

Subjects’ interaction is the core of most human activities. This is the reason why a lack of coordination is often the cause of missing goals, more than individual failure. While there are different subjective and objective measures to assess the level of mental effort required by subjects while facing a situation that is getting harder, that is, mental workload, to define an objective measure based on how and if team members are interacting is not so straightforward. In this study, behavioral, subjective and synchronized electroencephalographic data were collected from couples involved in a cooperative task to describe the relationship between task difficulty and team coordination, in the sense of interaction aimed at cooperatively performing the assignment. Multiple-brain connectivity analysis provided information about the whole interacting system. The results showed that averaged local properties of a brain network were affected by task difficulty. In particular, strength changed significantly with task difficulty and clustering coefficients strongly correlated with the workload itself. In particular, a higher workload corresponded to lower clustering values over the central and parietal brain areas. Such results has been interpreted as less efficient organization of the network when the subjects’ activities, due to high workload tendencies, were less coordinated

Capturing Complex Behavior in Brain Imaging: Strategies and Instrumentation

Functional neuroimaging investigates the human brain through non-invasive recordings of brain signals or non-invasive stimulation. Traditionally, neuroimaging practitioners attempted to restrict the subject's behavior throughout the experiment to the point where it could be completely characterized by a few simple variables. Although this approach has its merits, it considerably limits the possibilities for investigating neural mechanisms underlying the organism's function under natural conditions. To overcome this limitation, researchers have increasingly focused on neuroimaging studies of subjects involved in complex ecologically-valid behavioral tasks. The shift from simple to complex behavior in neuroimaging studies brings along the demand for: (1) new instrumentation for handling the behavioral aspect of the experiment, and (2) new experimental designs that exploit the complexity of the participant's behavior instead of trying to suppress it.  The thesis comprises four publications that examine the capacity of video technology to provide new instrumentation and explore possibilities for new experimental designs utilizing rich behavioural information provided by video, in the context of magnetoencephalography (MEG) and transcranial magnetic stimulation (TMS) methods. Additionally, it introduces the Helsinki VideoMEG Project an open-source collaborative effort aimed at providing MEG practitioners with video recording and analysis tools.  The first part of the thesis (Publications I and II) examines the feasibility of augmenting TMS and MEG experiments with simultaneous synchronized video and audio recordings of the participant. The second part of the thesis (Publications III and IV) explores the possibility of using audio and video to link the participants in an MEG hyperscanning experiment simultaneous recording of MEG signals from two interacting subjects.  The results presented in this thesis demonstrate the feasibility of augmenting TMS and MEG experiments with synchronized video and audio recordings

Multiple-Brain connectivity during third party punishment: an EEG hyperscanning study

Compassion is a particular form of empathic reaction to harm that befalls others and is accompanied by a desire to alleviate their suffering. This altruistic behavior is often manifested through altruistic punishment, wherein individuals penalize a deprecated human's actions, even if they are directed toward strangers. By adopting a dual approach, we provide empirical evidence that compassion is a multifaceted prosocial behavior and can predict altruistic punishment. In particular, in this multiple-brain connectivity study in an EEG hyperscanning setting, compassion was examined during real-time social interactions in a third-party punishment (TPP) experiment. We observed that specific connectivity patterns were linked to behavioral and psychological intra- and interpersonal factors. Thus, our results suggest that an ecological approach based on simultaneous dual-scanning and multiple-brain connectivity is suitable for analyzing complex social phenomena

Systemic physiology augmented functional near-infrared spectroscopy hyperscanning: a first evaluation investigating entrainment of spontaneous activity of brain and body physiology between subjects.

Significance: Functional near-infrared spectroscopy (fNIRS) enables measuring the brain activity of two subjects while they interact, i.e., the hyperscanning approach. Aim: In our exploratory study, we extended classical fNIRS hyperscanning by adding systemic physiological measures to obtain systemic physiology augmented fNIRS (SPA-fNIRS) hyperscanning while blocking and not blocking the visual communication between the subjects. This approach enables access brain-to-brain, brain-to-body, and body-to-body coupling between the subjects simultaneously. Approach: Twenty-four pairs of subjects participated in the experiment. The paradigm consisted of two subjects that sat in front of each other and had their eyes closed for 10 min, followed by a phase of 10 min where they made eye contact. Brain and body activity was measured continuously by SPA-fNIRS. Results: Our study shows that making eye contact for a prolonged time causes significant changes in brain-to-brain, brain-to-body, and body-to-body coupling, indicating that eye contact is followed by entrainment of the physiology between subjects. Subjects that knew each other generally showed a larger trend to change between the two conditions. Conclusions: The main point of this study is to introduce a new framework to investigate brain-to-brain, body-to-body, and brain-to-body coupling through a simple social experimental paradigm. The study revealed that eye contact leads to significant synchronization of spontaneous activity of the brain and body physiology. Our study is the first that employed the SPA-fNIRS approach and showed its usefulness to investigate complex interpersonal physiological changes

How Two Brains Make One Synchronized Mind in the Inferior Frontal Cortex: fNIRS-Based Hyperscanning During Cooperative Singing.

One form of communication that is common in all cultures is people singing together. Singing together reflects an index of cognitive synchronization and cooperation of human brains. Little is known about the neural synchronization mechanism, however. Here, we examined how two brains make one synchronized behavior using cooperated singing/humming between two people and hyperscanning, a new brain scanning technique. Hyperscanning allowed us to observe dynamic cooperation between interacting participants. We used functional near-infrared spectroscopy (fNIRS) to simultaneously record the brain activity of two people while they cooperatively sang or hummed a song in face-to-face (FtF) or face-to-wall (FtW) conditions. By calculating the inter-brain wavelet transform coherence between two interacting brains, we found a significant increase in the neural synchronization of the left inferior frontal cortex (IFC) for cooperative singing or humming regardless of FtF or FtW compared with singing or humming alone. On the other hand, the right IFC showed an increase in neural synchronization for humming only, possibly due to more dependence on musical processing

Quantification of inter-brain coupling: A review of current methods used in haemodynamic and electrophysiological hyperscanning studies

Hyperscanning is a form of neuroimaging experiment where the brains of two or more participants are imaged simultaneously whilst they interact. Within the domain of social neuroscience, hyperscanning is increasingly used to measure inter-brain coupling (IBC) and explore how brain responses change in tandem during social interaction. In addition to cognitive research, some have suggested that quantification of the interplay between interacting participants can be used as a biomarker for a variety of cognitive mechanisms aswell as to investigate mental health and developmental conditions including schizophrenia, social anxiety and autism. However, many different methods have been used to quantify brain coupling and this can lead to questions about comparability across studies and reduce research reproducibility. Here, we review methods for quantifying IBC, and suggest some ways moving forward. Following the PRISMA guidelines, we reviewed 215 hyperscanning studies, across four different brain imaging modalities: functional near-infrared spectroscopy (fNIRS), functional magnetic resonance (fMRI), electroencephalography (EEG) and magnetoencephalography (MEG). Overall, the review identified a total of 27 different methods used to compute IBC. The most common hyperscanning modality is fNIRS, used by 119 studies, 89 of which adopted wavelet coherence. Based on the results of this literature survey, we first report summary statistics of the hyperscanning field, followed by a brief overview of each signal that is obtained from each neuroimaging modality used in hyperscanning. We then discuss the rationale, assumptions and suitability of each method to different modalities which can be used to investigate IBC. Finally, we discuss issues surrounding the interpretation of each method

Biomedical Signal Analysis of the Brain and Systemic Physiology

Near-infrared spectroscopy (NIRS) is a non-invasive and easy-to-use diagnostic technique that enables real-time tissue oxygenation measurements applied in various contexts and for different purposes. Continuous monitoring with NIRS of brain oxygenation, for example, in neonatal intensive care units (NICUs), is essential to prevent lifelong disabilities in newborns. Moreover, NIRS can be applied to observe brain activity associated with hemodynamic changes in blood flow due to neurovascular coupling. In the latter case, NIRS contributes to studying cognitive processes allowing to conduct experiments in natural and socially interactive contexts of everyday life. However, it is essential to measure systemic physiology and NIRS signals concurrently. The combination of brain and body signals enables to build sophisticated systems that, for example, reduce the false alarms that occur in NICUs. Furthermore, since fNIRS signals are influenced by systemic physiology, it is essential to understand how the latter impacts brain signals in functional studies. There is an interesting brain body coupling that has rarely been investigated yet. To take full advantage of these brain and body data, the aim of this thesis was to develop novel approaches to analyze these biosignals to extract the information and identify new patterns, to solve different research or clinical questions. For this the development of new methodological approaches and sophisticated data analysis is necessary, because often the identification of these patterns is challenging or not possible with traditional methods. In such cases, automatic machine learning (ML) techniques are beneficial. The first contribution of this work was to assess the known systemic physiology augmented (f)NIRS approach for clinical use and in everyday life. Based on physiological and NIRS signals of preterm infants, an ML-based classification system has been realized, able to reduce the false alarms in NICUs by providing a high sensitivity rate. In addition, the SPA-fNIRS approach was further applied in adults during a breathing task. The second contribution of this work was the advancement of the classical fNIRS hyperscanning method by adding systemic physiology measures. For this, new biosignal analyses in the time-frequency domain have been developed and tested in a simple nonverbal synchrony task between pairs of subjects. Furthermore, based on SPA-fNIRS hyperscanning data, another ML-based system was created, which is able distinguish familiar and unfamiliar pairs with high accuracy. This approach enables to determine the strength of social bonds in a wide range of social interaction contexts. In conclusion, we were the first group to perform a SPA-fNIRS hyperscanning study capturing changes in cerebral oxygenation and hemodynamics as well as systemic physiology in two subjects simultaneously. We applied new biosignals analysis methods enabling new insights into the study of social interactions. This work opens the door to many future inter-subjects fNIRS studies with the benefit of assessing the brain-to-brain, the brain-to-body, and body-to-body coupling between pairs of subjects

Multivariate model for cooperation: bridging Social Physiological Compliance and Hyperscanning

The neurophysiological analysis of cooperation has evolved over the past 20 years, moving towards the research of common patterns in neurophysiological signals of people interacting. Social Physiological Compliance (SPC) and Hyperscanning represent two frameworks for the joint analysis of autonomic and brain signals respectively. Each of the two approaches allows to know about a single layer of cooperation according to the nature of these signals: SPC provides information mainly related to emotions, and Hyperscanning that related to cognitive aspects. In this work, after the analysis of the state of the art of SPC and Hyperscanning, we explored the possibility to unify the two approaches creating a complete neurophysiological model for cooperation considering both affective and cognitive mechanisms. We synchronously recorded electrodermal activity, cardiac and brain signals of 14 cooperative dyads. Time series from these signals were extracted and Multivariate Granger Causality was computed. The results showed that only when subjects in a dyad cooperate there is a statistically significant causality between the multivariate variables representing each subject. Moreover, the entity of this statistical relationship correlates with the dyad's performance. Finally, given the novelty of this approach and its exploratory nature, we provided its strengths and limitations

A novel approach to measure brain-to-brain spatial and temporal alignment during positive empathy

: Empathy is defined as the ability to vicariously experience others' suffering (vicarious pain) or feeling their joy (vicarious reward). While most neuroimaging studies have focused on vicarious pain and describe similar neural responses during the observed and the personal negative affective involvement, only initial evidence has been reported for the neural responses to others' rewards and positive empathy. Here, we propose a novel approach, based on the simultaneous recording of multi-subject EEG signals and exploiting the wavelet coherence decomposition to measure the temporal alignment between ERPs in a dyad of interacting subjects. We used the Third-Party Punishment (TPP) paradigm to elicit the personal and vicarious experiences. During a positive experience, we observed the simultaneous presence in both agents of the Late Positive Potential (LPP), an ERP component related to emotion processing, as well as the existence of an inter-subject ERPs synchronization in the related time window. Moreover, the amplitude of the LPP synchronization was modulated by the presence of a human-agent. Finally, the localized brain circuits subtending the ERP-synchronization correspond to key-regions of personal and vicarious reward. Our findings suggest that the temporal and spatial ERPs alignment might be a novel and direct proxy measure of empathy