Lipreading a naturalistic narrative in a female population : Neural characteristics shared with listening and reading

Publisher Copyright: © 2022 The Authors. Brain and Behavior published by Wiley Periodicals LLC.Introduction: Few of us are skilled lipreaders while most struggle with the task. Neural substrates that enable comprehension of connected natural speech via lipreading are not yet well understood. Methods: We used a data-driven approach to identify brain areas underlying the lipreading of an 8-min narrative with participants whose lipreading skills varied extensively (range 6–100%, mean = 50.7%). The participants also listened to and read the same narrative. The similarity between individual participants’ brain activity during the whole narrative, within and between conditions, was estimated by a voxel-wise comparison of the Blood Oxygenation Level Dependent (BOLD) signal time courses. Results: Inter-subject correlation (ISC) of the time courses revealed that lipreading, listening to, and reading the narrative were largely supported by the same brain areas in the temporal, parietal and frontal cortices, precuneus, and cerebellum. Additionally, listening to and reading connected naturalistic speech particularly activated higher-level linguistic processing in the parietal and frontal cortices more consistently than lipreading, probably paralleling the limited understanding obtained via lip-reading. Importantly, higher lipreading test score and subjective estimate of comprehension of the lipread narrative was associated with activity in the superior and middle temporal cortex. Conclusions: Our new data illustrates that findings from prior studies using well-controlled repetitive speech stimuli and stimulus-driven data analyses are also valid for naturalistic connected speech. Our results might suggest an efficient use of brain areas dealing with phonological processing in skilled lipreaders.Peer reviewe

Brain-to-brain hyperclassification reveals action-specific motor mapping of observed actions in humans

Seeing an action may activate the corresponding action motor code in the observer. It remains unresolved whether seeing and performing an action activates similar action-specific motor codes in the observer and the actor. We used novel hyperclassification approach to reveal shared brain activation signatures of action execution and observation in interacting human subjects. In the first experiment, two "actors" performed four types of hand actions while their haemodynamic brain activations were measured with 3-T functional magnetic resonance imaging (fMRI). The actions were videotaped and shown to 15 "observers" during a second fMRI experiment. Eleven observers saw the videos of one actor, and the remaining four observers saw the videos of the other actor. In a control fMRI experiment, one of the actors performed actions with closed eyes, and five new observers viewed these actions. Bayesian canonical correlation analysis was applied to functionally realign observers' and actors' fMRI data. Hyperclassification of the seen actions was performed with Bayesian logistic regression trained on actors' data and tested with observers' data. Without the functional realignment, between-subjects accuracy was at chance level. With the realignment, the accuracy increased on average by 15 percentage points, exceeding both the chance level and the accuracy without functional realignment. The highest accuracies were observed in occipital, parietal and premotor cortices. Hyperclassification exceeded chance level also when the actor did not see her own actions. We conclude that the functional brain activation signatures underlying action execution and observation are partly shared, yet these activation signatures may be anatomically misaligned across individuals

Stimulus-Related Independent Component and Voxel-Wise Analysis of Human Brain Activity during Free Viewing of a Feature Film

Understanding how the brain processes stimuli in a rich natural environment is a fundamental goal of neuroscience. Here, we showed a feature film to 10 healthy volunteers during functional magnetic resonance imaging (fMRI) of hemodynamic brain activity. We then annotated auditory and visual features of the motion picture to inform analysis of the hemodynamic data. The annotations were fitted to both voxel-wise data and brain network time courses extracted by independent component analysis (ICA). Auditory annotations correlated with two independent components (IC) disclosing two functional networks, one responding to variety of auditory stimulation and another responding preferentially to speech but parts of the network also responding to non-verbal communication. Visual feature annotations correlated with four ICs delineating visual areas according to their sensitivity to different visual stimulus features. In comparison, a separate voxel-wise general linear model based analysis disclosed brain areas preferentially responding to sound energy, speech, music, visual contrast edges, body motion and hand motion which largely overlapped the results revealed by ICA. Differences between the results of IC- and voxel-based analyses demonstrate that thorough analysis of voxel time courses is important for understanding the activity of specific sub-areas of the functional networks, while ICA is a valuable tool for revealing novel information about functional connectivity which need not be explained by the predefined model. Our results encourage the use of naturalistic stimuli and tasks in cognitive neuroimaging to study how the brain processes stimuli in rich natural environments

Naturalizing psychopathology—towards a quantitative real-world psychiatry

Psychiatric disorders continue to be on the rise around the globe. Meanwhile, efforts and investments directed to early diagnosis and appropriate interventions for mental health problems are lagging resulting in ‘substantial loss of human capabilities and avoidable suffering’ [1]. A major component of our inability to address mental health problems resides in the persistent lack of objective measures for evaluating such difficulties in the daily life of individuals, complicating the detection of clinically relevant changes in the patients’ well-being [2]. Similarly, most studies into the neurobiology of psychiatric disorders lack a detailed description of individual functioning despite influential calls for quantitative approaches to psychopathology

Unobtrusive tracking of interpersonal orienting and distance predicts the subjective quality of social interactions

Interpersonal coordination of behaviour is essential forsmooth social interactions. Measures of interpersonal behaviour,however, often rely on subjective evaluations, invasivemeasurement techniques or gross measures of motion. Here,we constructed an unobtrusive motion tracking system thatenables detailed analysis of behaviour at the individual andinterpersonal levels, which we validated using wearablesensors. We evaluate dyadic measures of joint orienting anddistancing, synchrony and gaze behaviours to summarize datacollected during natural conversation and joint action tasks.Our results demonstrate that patterns of proxemic behaviours,rather than more widely used measures of interpersonalsynchrony, best predicted the subjective quality of theinteractions. Increased distance between participants predictedlower enjoyment, while increased joint orienting towards eachother during cooperation correlated with increased effortreported by the participants. Importantly, the interpersonaldistance was most informative of the quality of interactionwhen task demands and experimental control were minimal.gathered during minimally constrained social interactions are particularly sensitive for the subjectivequality of social interactions and may be useful for interaction-based phenotyping for further studies

Interpersonal similarity of autistic traits predicts friendship quality

Autistic traits are known to be associated with social interaction difficulties. Yet, somewhat paradoxically, relevant research has been typically restricted to studying individuals. In line with the ‘dialectical misattunement hypothesis’ and clinical insights of intact social interactions among autistic individuals, we hypothesized that friendship quality varies as a function of interpersonal similarity and more concretely the difference value of autistic traits in a dyad, above and beyond autistic traits per se. Therefore, in this study, we used self-report questionnaires to investigate these measures in a sample of 67 neurotypical dyads across a broad range of autistic traits. Our results demonstrate that the more similar two persons are in autistic traits, the higher is the perceived quality of their friendship, irrespective of friendship duration, age, sex and, importantly, the (average of) autistic traits in a given dyad. More specifically, higher interpersonal similarity of autistic traits was associated with higher measures of closeness, acceptance and help. These results, therefore, lend support to the idea of an interactive turn in the study of social abilities across the autism spectrum and pave the way for future studies on the multiscale dynamics of social interactions

Fronto-parietal network supports context-dependent speech comprehension

Knowing the context of a discourse is an essential prerequisite for comprehension. Here we used functional magnetic resonance imaging (fMRI) to disclose brain networks supporting context-dependent speech comprehension. During fMRI, 20 participants listened to 1-min spoken narratives preceded by pictures that were either contextually matching or mismatching with the narrative. Matching pictures increased narrative comprehension, decreased hemodynamic activity in Broca׳s area, and enhanced its functional connectivity with left anterior superior frontal gyrus, bilateral inferior parietal cortex, as well as anterior and posterior cingulate cortex. Further, the anterior (BA 45) and posterior (BA 44) portions of Broca׳s area differed in their functional connectivity patterns. Both BA 44 and BA 45 have shown increased connectivity with right angular gyrus and supramarginal gyrus. Whereas BA 44 showed increased connectivity with left angular gyrus, left inferior/middle temporal gyrus and left postcentral gyrus, BA 45showed increased connectivity with right posterior cingulate cortex, right anterior inferior frontal gyrus, lateral occipital cortex and anterior cingulate cortex. Our results suggest that a fronto-parietal functional network supports context-dependent narrative comprehension, and that Broca׳s area is involved in resolving ambiguity from speech when appropriate contextual cues are lacking.Peer reviewe

Differential inter-subject correlation of brain activity when kinship is a variable in moral dilemma

Previous behavioural studies have shown that humans act more altruistically towards kin. Whether and how knowledge of genetic relatedness translates into differential neurocognitive evaluation of observed social interactions has remained an open question. Here, we investigated how the human brain is engaged when viewing a moral dilemma between genetic vs. non-genetic sisters. During functional magnetic resonance imaging, a movie was shown, depicting refusal of organ donation between two sisters, with subjects guided to believe the sisters were related either genetically or by adoption. Although 90% of the subjects self-reported that genetic relationship was not relevant, their brain activity told a different story. Comparing correlations of brain activity across all subject pairs between the two viewing conditions, we found significantly stronger inter-subject correlations in insula, cingulate, medial and lateral prefrontal, superior temporal, and superior parietal cortices, when the subjects believed that the sisters were genetically related. Cognitive functions previously associated with these areas include moral and emotional conflict regulation, decision making, and mentalizing, suggesting more similar engagement of such functions when observing refusal of altruism from a genetic sister. Our results show that mere knowledge of a genetic relationship between interacting persons robustly modulates social cognition of the perceiver.Peer reviewe

Areas showing significant correlations with single auditory and visual features.

<p>The color coding as in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0035215#pone-0035215-g008" target="_blank">Figure 8</a>. Results are thresholded at p < 0.001. A: Speech explains activity in the superior temporal sulcus (STS) and middle temporal gyrus (MTG), lip representation area of the primary motor cortex (M1L) and ptIFG particularly in the left hemisphere, and dorsomedial PFC. Partially overlapping areas also show activity correlated with music. RMS energy explains activity in the superior temporal areas, particularly in a part of the posterior bank of Heschl’s gyrus (in or in the vicinity of the primary auditory cortex) and/or Planum Temporale. Other sound categories are not significantly correlated with activity in this region. The black outline indicates the area encompassing the Heschl’s gyrus (HG) of all subjects visually identified from the standardized structural images. Bar graphs show the correlation coefficients for each stimulus feature in the non-overlapping areas. Colors of the bars refer to the brain areas best activated with one of the three stimulus features. B: Hand motion activates strongly IPS and TOJ. The areas are very similar to those included in IC6 (see <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0035215#pone-0035215-g007" target="_blank">Figure 7</a>). Superior occipital cortex (sOC), occipital pole (OP), and parts of the lateral occipital cortex (lOC) show specific activity to body motion. Activity in the occipital pole also correlated with the contrast edges of the image.</p