Neural processing of semantic content in movies

Naturalistic stimuli, such as movies, contain interacting, multimodal and semantic features and allow for free exploration through eye movements. The full extent of neural responses to features such as motion, film cuts and eye movement behavior has not been established. The main hypothesis of this thesis is that complex multimodal and semantic stimuli in naturalistic movies engage a widespread ensemble of locations across the entire brain. To address this question I analyzed simultaneous intracranial and eyetracking data from over 6,000 electrodes across 23 patients with intractable epilepsy. Responses to fast eye movements – saccades – and film cuts are widespread across the entire brain, while responses to motion are restricted to visual brain areas. Higher-order brain areas respond differentially to semantic and low-level changes across film cuts and saccades. Movies have also recently been used in combination with resting state scans to investigate the utility of functional connectivity as a potential biomarker for psychiatric disorders. Functional connectivity in fMRI data measured during resting state and movie conditions is reliable, subject-specific and related to phenotype. However, it is unclear whether functional connectivity of EEG also possesses these qualities, which are required for the clinical use of neural biomarkers. I hypothesize that functional connectivity networks measured in EEG data recorded during movie watching are a predictor of psychiatric phenotypes similar to functional connectivity of fMRI. I demonstrate that functional connectivity of EEG is reliable, subject-specific and related to phenotypes. However, the patterns of functional connectivity differ in EEG and fMRI, suggesting the measures capture complementary information. In summary, these results demonstrate that the semantic content in movies allows one to study neural processing in naturalistic settings. In addition, EEG functional connectivity recorded during resting state and movie condition is reliabe, subject-specific and related to phenotype

Saccadic modulation of neural excitability in auditory areas of the neocortex

In natural "active" vision, humans and other primates use eye movements (saccades) to sample bits of information from visual scenes. In the visual cortex, non-retinal signals linked to saccades shift visual cortical neurons into a high excitability state as each saccade ends. The extent of this saccadic modulation outside of the visual system is unknown. Here, we show that during natural viewing, saccades modulate excitability in numerous auditory cortical areas with a temporal pattern complementary to that seen in visual areas. Control somatosensory cortical recordings indicate that the temporal pattern is unique to auditory areas. Bidirectional functional connectivity patterns suggest that these effects may arise from regions involved in saccade generation. We propose that by using saccadic signals to yoke excitability states in auditory areas to those in visual areas, the brain can improve information processing in complex natural settings

Functional connectivity of EEG is subject-specific, associated with phenotype, and different from fMRI

A variety of psychiatric, behavioral and cognitive phenotypes have been linked to brain ‘’functional connectivity’’ -- the pattern of correlation observed between different brain regions. Most commonly assessed using functional magnetic resonance imaging (fMRI), here, we investigate the connectivity-phenotype associations with functional connectivity measured with electroencephalography (EEG), using phase-coupling. We analyzed data from the publicly available Healthy Brain Network Biobank. This database compiles a growing sample of children and adolescents, currently encompassing 1657 individuals. Among a variety of assessment instruments we focus on ten phenotypic and additional demographic measures that capture most of the variance in this sample. The largest effect sizes are found for age and sex for both fMRI and EEG. We replicate previous findings of an association of Intelligence Quotient (IQ) and Attention Deficit Hyperactivity Disorder (ADHD) with the pattern of fMRI functional connectivity. We also find an association with socioeconomic status, anxiety and the Child Behavior Checklist Score. For EEG we find a significant connectivity-phenotype relationship with IQ. The actual spatial patterns of functional connectivity are quite different between fMRI and source-space EEG. However, within EEG we observe clusters of functional connectivity that are consistent across frequency bands. Additionally we analyzed reproducibility of functional connectivity. We compare connectivity obtained with different tasks, including resting state, a video and a visual flicker task. For both EEG and fMRI the variation between tasks was smaller than the variability observed between subjects. We also found an increase of reliability with increasing frequency of the EEG, and increased sampling duration. We conclude that, while the patterns of functional connectivity are distinct between fMRI and phase-coupling of EEG, they are nonetheless similar in their robustness to the task, and similar in that idiosyncratic patterns of connectivity predict individual phenotypes

Data - Semantic novelty modulates neural responses to visual change across the human brain

Data to reproduce statistical analysis and Figure corresponding to: https://doi.org/10.1101/2022.06.20.49646

An open-access dataset of naturalistic viewing using simultaneous EEG-fMRI

Abstract In this work, we present a dataset that combines functional magnetic imaging (fMRI) and electroencephalography (EEG) to use as a resource for understanding human brain function in these two imaging modalities. The dataset can also be used for optimizing preprocessing methods for simultaneously collected imaging data. The dataset includes simultaneously collected recordings from 22 individuals (ages: 23–51) across various visual and naturalistic stimuli. In addition, physiological, eye tracking, electrocardiography, and cognitive and behavioral data were collected along with this neuroimaging data. Visual tasks include a flickering checkerboard collected outside and inside the MRI scanner (EEG-only) and simultaneous EEG-fMRI recordings. Simultaneous recordings include rest, the visual paradigm Inscapes, and several short video movies representing naturalistic stimuli. Raw and preprocessed data are openly available to download. We present this dataset as part of an effort to provide open-access data to increase the opportunity for discoveries and understanding of the human brain and evaluate the correlation between electrical brain activity and blood oxygen level-dependent (BOLD) signals