Optimizing methods for linking cinematic features to fMRI data

One of the challenges of naturalistic neurosciences using movie-viewing experiments is how to interpret observed brain activations in relation to the multiplicity of time-locked stimulus features. As previous studies have shown less inter-subject synchronization across viewers of random video footage than story-driven films, new methods need to be developed for analysis of less story-driven contents. To optimize the linkage between our fMRI data collected during viewing of a deliberately non-narrative silent film 'At Land' by Maya Deren (1944) and its annotated content, we combined the method of elastic-net regularization with the model-driven linear regression and the well-established data-driven independent component analysis (ICA) and inter-subject correlation (ISC) methods. In the linear regression analysis, both IC and region-of-interest (ROI) time-series were fitted with time-series of a total of 36 binary-valued and one real-valued tactile annotation of film features. The elastic-net regularization and cross-validation were applied in the ordinary least-squares linear regression in order to avoid over-fitting due to the multicollinearity of regressors, the results were compared against both the partial least-squares (PLS) regression and the un-regularized full-model regression. Nonparametric permutation testing scheme was applied to evaluate the statistical significance of regression. We found statistically significant correlation between the annotation model and 9 ICs out of 40 ICs. Regression analysis was also repeated for a large set of cubic ROIs covering the grey matter. Both IC- and ROI-based regression analyses revealed activations in parietal and occipital regions, with additional smaller clusters in the frontal lobe. Furthermore, we found elastic-net based regression more sensitive than PLS and un-regularized regression since it detected a larger number of significant ICs and ROIs. Along with the ISC ranking methods, our regression analysis proved a feasible method for ordering the ICs based on their functional relevance to the annotated cinematic features. The novelty of our method is - in comparison to the hypothesis-driven manual pre-selection and observation of some individual regressors biased by choice - in applying data-driven approach to all content features simultaneously. We found especially the combination of regularized regression and ICA useful when analyzing fMRI data obtained using non-narrative movie stimulus with a large set of complex and correlated features. (C) 2015 The Authors. Published by Elsevier Inc.Peer reviewe

The Neurocognitive Process of Digital Radicalization: A Theoretical Model and Analytical Framework

Recent studies suggest that empathy induced by narrative messages can effectively facilitate persuasion and reduce psychological reactance. Although limited, emerging research on the etiology of radical political behavior has begun to explore the role of narratives in shaping an individual’s beliefs, attitudes, and intentions that culminate in radicalization. The existing studies focus exclusively on the influence of narrative persuasion on an individual, but they overlook the necessity of empathy and that in the absence of empathy, persuasion is not salient. We argue that terrorist organizations are strategic in cultivating empathetic-persuasive messages using audiovisual materials, and disseminating their message within the digital medium. Therefore, in this paper we propose a theoretical model and analytical framework capable of helping us better understand the neurocognitive process of digital radicalization

Narrative comprehension beyond language : Common brain networks activated by a movie and its script

Narratives surround us in our everyday life in different forms. In the sensory brain areas, the processing of narratives is dependent on the media of presentation, be that in audiovisual or written form. However, little is known of the brain areas that process complex narrative content mediated by various forms. To isolate these regions, we looked for the functional networks reacting in a similar manner to the same narrative content despite different media of presentation. We collected 3-T fMRI whole brain data from 31 healthy human adults during two separate runs when they were either viewing a movie or reading its screenplay text. The independent component analysis (ICA) was used to separate 40 components. By correlating the components' time-courses between the two different media conditions, we could isolate 5 functional networks that particularly related to the same narrative content. These TOP-5 components with the highest correlation covered fronto-temporal, parietal, and occipital areas with no major involvement of primary visual or auditory cortices. Interestingly, the top-ranked network with highest modality-invariance also correlated negatively with the dialogue predictor, thus pinpointing that narrative comprehension entails processes that are not language-reliant. In summary, our novel experiment design provided new insight into narrative comprehension networks across modalities.Peer reviewe

Processing of pragmatic communication in ASD: a video-based brain imaging study

Peer reviewe

Consistency and similarity of MEG- and fMRI-signal time courses during movie viewing

Movie viewing allows human perception and cognition to be studied in complex, real-life-like situations in a brain-imaging laboratory. Previous studies with functional magnetic resonance imaging (fMRI) and with magneto-and electroencephalography (MEG and EEG) have demonstrated consistent temporal dynamics of brain activity across movie viewers. However, little is known about the similarities and differences of fMRI and MEG or EEG dynamics during such naturalistic situations. We thus compared MEG and fMRI responses to the same 15-min black-and-white movie in the same eight subjects who watched the movie twice during both MEG and fMRI recordings. We analyzed intra-and intersubject voxel-wise correlations within each imaging modality as well as the correlation of the MEG envelopes and fMRI signals. The fMRI signals showed voxel-wise within-and between-subjects correlations up to r = 0.66 and r = 0.37, respectively, whereas these correlations were clearly weaker for the envelopes of band-pass filtered (7 frequency bands below 100 Hz) MEG signals (within-subjects correlation r <0.14 and between-subjects r <0.05). Direct MEG-fMRI voxel-wise correlations were unreliable. Notably, applying a spatial-filtering approach to the MEG data uncovered consistent canonical variates that showed considerably stronger (up to r = 0.25) between-subjects correlations than the univariate voxel-wise analysis. Furthermore, the envelopes of the time courses of these variates up to about 10 Hz showed association with fMRI signals in a general linear model. Similarities between envelopes of MEG canonical variates and fMRI voxel time-courses were seen mostly in occipital, but also in temporal and frontal brain regions, whereas intra-and intersubject correlations for MEG and fMRI separately were strongest only in the occipital areas. In contrast to the conventional univariate analysis, the spatial-filtering approach was able to uncover associations between the MEG envelopes and fMRI time courses, shedding light on the similarities of hemodynamic and electromagnetic brain activities during movie viewing.Peer reviewe

High-Density Diffuse Optical Tomography During Passive Movie Viewing: A Platform for Naturalistic Functional Brain Mapping

Human neuroimaging techniques enable researchers and clinicians to non-invasively study brain function across the lifespan in both healthy and clinical populations. However, functional brain imaging methods such as functional magnetic resonance imaging (fMRI) are expensive, resource-intensive, and require dedicated facilities, making these powerful imaging tools generally unavailable for assessing brain function in settings demanding open, unconstrained, and portable neuroimaging assessments. Tools such as functional near-infrared spectroscopy (fNIRS) afford greater portability and wearability, but at the expense of cortical field-of-view and spatial resolution. High-Density Diffuse Optical Tomography (HD-DOT) is an optical neuroimaging modality directly addresses the image quality limitations associated with traditional fNIRS techniques through densely overlapping optical measurements. This thesis aims to establish the feasibility of using HD-DOT in a novel application demanding exceptional portability and flexibility: mapping disrupted cortical activity in chronically malnourished children. I first motivate the need for dense optical measurements of brain tissue to achieve fMRI-comparable localization of brain function (Chapter 2). Then, I present imaging work completed in Cali, Colombia, where a cohort of chronically malnourished children were imaged using a custom HD-DOT instrument to establish feasibility of performing field-based neuroimaging in this population (Chapter 3). Finally, in order to meet the need for age appropriate imaging paradigms in this population, I develop passive movie viewing paradigms for use in optical neuroimaging, a flexible and rich stimulation paradigm that is suitable for both adults and children (Chapter 4)

Integrating media content analysis, reception analysis, and media effects studies

Every day, the world of media is at our fingertips, whether it is watching movies, listening to the radio, or browsing online media. On average, people spend over 8 h per day consuming messages from the mass media, amounting to a total lifetime dose of more than 20 years in which conceptual content stimulates our brains. Effects from this flood of information range from short-term attention bursts (e.g., by breaking news features or viral ‘memes’) to life-long memories (e.g., of one’s favorite childhood movie), and from micro-level impacts on an individual’s memory, attitudes, and behaviors to macro-level effects on nations or generations. The modern study of media’s influence on society dates back to the 1940s. This body of mass communication scholarship has largely asked, “what is media’s effect on the individual?” Around the time of the cognitive revolution, media psychologists began to ask, “what cognitive processes are involved in media processing?” More recently, neuroimaging researchers started using real-life media as stimuli to examine perception and cognition under more natural conditions. Such research asks: “what can media tell us about brain function?” With some exceptions, these bodies of scholarship often talk past each other. An integration offers new insights into the neurocognitive mechanisms through which media affect single individuals and entire audiences. However, this endeavor faces the same challenges as all interdisciplinary approaches: Researchers with different backgrounds have different levels of expertise, goals, and foci. For instance, neuroimaging researchers label media stimuli as “naturalistic” although they are in many ways rather artificial. Similarly, media experts are typically unfamiliar with the brain. Neither media creators nor neuroscientifically oriented researchers approach media effects from a social scientific perspective, which is the domain of yet another species. In this article, we provide an overview of approaches and traditions to studying media, and we review the emerging literature that aims to connect these streams. We introduce an organizing scheme that connects the causal paths from media content → brain responses → media effects and discuss network control theory as a promising framework to integrate media content, reception, and effects analyses

A Model of the Network Architecture of the Brain that Supports Natural Language Processing

For centuries, neuroscience has proposed models of the neurobiology of language processing that are static and localised to few temporal and inferior frontal regions. Although existing models have offered some insight into the processes underlying lower-level language features, they have largely overlooked how language operates in the real world. Here, we aimed at investigating the network organisation of the brain and how it supports language processing in a naturalistic setting. We hypothesised that the brain is organised in a multiple core-periphery and dynamic modular architecture, with canonical language regions forming high-connectivity hubs. Moreover, we predicted that language processing would be distributed to much of the rest of the brain, allowing it to perform more complex tasks and to share information with other cognitive domains. To test these hypotheses, we collected the Naturalistic Neuroimaging Database of people watching full length movies during functional magnetic resonance imaging. We computed network algorithms to capture the voxel-wise architecture of the brain in individual participants and inspected variations in activity distribution over different stimuli and over more complex language features. Our results confirmed the hypothesis that the brain is organised in a flexible multiple core-periphery architecture with large dynamic communities. Here, language processing was distributed to much of the rest of the brain, together forming multiple communities. Canonical language regions constituted hubs, explaining why they consistently appear in various other neurobiology of language models. Moreover, language processing was supported by other regions such as visual cortex and episodic memory regions, when processing more complex context-specific language features. Overall, our flexible and distributed model of language comprehension and the brain points to additional brain regions and pathways that could be exploited for novel and more individualised therapies for patients suffering from speech impairments

High-Density Diffuse Optical Tomography During Passive Movie Viewing: A Platform for Naturalistic Functional Brain Mapping

Human neuroimaging techniques enable researchers and clinicians to non-invasively study brain function across the lifespan in both healthy and clinical populations. However, functional brain imaging methods such as functional magnetic resonance imaging (fMRI) are expensive, resource-intensive, and require dedicated facilities, making these powerful imaging tools generally unavailable for assessing brain function in settings demanding open, unconstrained, and portable neuroimaging assessments. Tools such as functional near-infrared spectroscopy (fNIRS) afford greater portability and wearability, but at the expense of cortical field-of-view and spatial resolution. High-Density Diffuse Optical Tomography (HD-DOT) is an optical neuroimaging modality directly addresses the image quality limitations associated with traditional fNIRS techniques through densely overlapping optical measurements. This thesis aims to establish the feasibility of using HD-DOT in a novel application demanding exceptional portability and flexibility: mapping disrupted cortical activity in chronically malnourished children. I first motivate the need for dense optical measurements of brain tissue to achieve fMRI-comparable localization of brain function (Chapter 2). Then, I present imaging work completed in Cali, Colombia, where a cohort of chronically malnourished children were imaged using a custom HD-DOT instrument to establish feasibility of performing field-based neuroimaging in this population (Chapter 3). Finally, in order to meet the need for age appropriate imaging paradigms in this population, I develop passive movie viewing paradigms for use in optical neuroimaging, a flexible and rich stimulation paradigm that is suitable for both adults and children (Chapter 4)