35 research outputs found
a methodological approach
In natural environments, visual and auditory stimulation elicit responses
across a large set of brain regions in a fraction of a second, yielding
representations of the multimodal scene and its properties. The rapid and
complex neural dynamics underlying visual and auditory information processing
pose major challenges to human cognitive neuroscience. Brain signals measured
non-invasively are inherently noisy, the format of neural representations is
unknown, and transformations between representations are complex and often
nonlinear. Further, no single non-invasive brain measurement technique
provides a spatio-temporally integrated view. In this opinion piece, we argue
that progress can be made by a concerted effort based on three pillars of
recent methodological development: (i) sensitive analysis techniques such as
decoding and cross-classification, (ii) complex computational modelling using
models such as deep neural networks, and (iii) integration across imaging
methods (magnetoencephalography/electroencephalography, functional magnetic
resonance imaging) and models, e.g. using representational similarity
analysis. We showcase two recent efforts that have been undertaken in this
spirit and provide novel results about visual and auditory scene analysis.
Finally, we discuss the limits of this perspective and sketch a concrete
roadmap for future research
Coherent categorical information triggers integration-related alpha dynamics
To create coherent visual experiences, the brain spatially integrates the complex and dynamic information it receives from the environment. We previously demonstrated that feedback-related alpha activity carries stimulus-specific information when two spatially and temporally coherent naturalistic inputs can be integrated into a unified percept. In this study, we sought to determine whether such integration-related alpha dynamics are triggered by categorical coherence in visual inputs. In an EEG experiment, we manipulated the degree of coherence by presenting pairs of videos from the same or different categories through two apertures in the left and right visual hemifields. Critically, video pairs could be video-level coherent (i.e., stem from the same video), coherent in their basic-level category, coherent in their superordinate category, or incoherent (i.e., stem from videos from two entirely different categories). We conducted multivariate classification analyses on rhythmic EEG responses to decode between the video stimuli in each condition. As the key result, we significantly decoded the video-level coherent and basic-level coherent stimuli, but not the superordinate coherent and incoherent stimuli, from cortical alpha rhythms. This suggests that alpha dynamics play a critical role in integrating information across space, and that cortical integration processes are flexible enough to accommodate information from different exemplars of the same basic-level category.
NEW & NOTEWORTHY Our brain integrates dynamic inputs across the visual field to create coherent visual experiences. Such integration processes have previously been linked to cortical alpha dynamics. In this study, the integration-related alpha activity was observed not only when snippets from the same video were presented, but also when different video snippets from the same basic-level category were presented, highlighting the flexibility of neural integration processes
Communicative signals during joint attention promote neural processes of infants and caregivers
Communicative signals such as eye contact increase infants’ brain activation to visual stimuli and promote joint attention. Our study assessed whether communicative signals during joint attention enhance infant-caregiver dyads’ neural responses to objects, and their neural synchrony. To track mutual attention processes, we applied rhythmic visual stimulation (RVS), presenting images of objects to 12-month-old infants and their mothers (n = 37 dyads), while we recorded dyads’ brain activity (i.e., steady-state visual evoked potentials, SSVEPs) with electroencephalography (EEG) hyperscanning. Within dyads, mothers either communicatively showed the images to their infant or watched the images without communicative engagement. Communicative cues increased infants’ and mothers’ SSVEPs at central-occipital-parietal, and central electrode sites, respectively. Infants showed significantly more gaze behaviour to images during communicative engagement. Dyadic neural synchrony (SSVEP amplitude envelope correlations, AECs) was not modulated by communicative cues. Taken together, maternal communicative cues in joint attention increase infants’ neural responses to objects, and shape mothers’ own attention processes. We show that communicative cues enhance cortical visual processing, thus play an essential role in social learning. Future studies need to elucidate the effect of communicative cues on neural synchrony during joint attention. Finally, our study introduces RVS to study infant-caregiver neural dynamics in social contexts
A multivariate comparison of electroencephalogram and functional magnetic resonance imaging to electrocorticogram using visual object representations in humans
Today, most neurocognitive studies in humans employ the non-invasive neuroimaging techniques functional magnetic resonance imaging (fMRI) and electroencephalogram (EEG). However, how the data provided by fMRI and EEG relate exactly to the underlying neural activity remains incompletely understood. Here, we aimed to understand the relation between EEG and fMRI data at the level of neural population codes using multivariate pattern analysis. In particular, we assessed whether this relation is affected when we change stimuli or introduce identity-preserving variations to them. For this, we recorded EEG and fMRI data separately from 21 healthy participants while participants viewed everyday objects in different viewing conditions, and then related the data to electrocorticogram (ECoG) data recorded for the same stimulus set from epileptic patients. The comparison of EEG and ECoG data showed that object category signals emerge swiftly in the visual system and can be detected by both EEG and ECoG at similar temporal delays after stimulus onset. The correlation between EEG and ECoG was reduced when object representations tolerant to changes in scale and orientation were considered. The comparison of fMRI and ECoG overall revealed a tighter relationship in occipital than in temporal regions, related to differences in fMRI signal-to-noise ratio. Together, our results reveal a complex relationship between fMRI, EEG, and ECoG signals at the level of population codes that critically depends on the time point after stimulus onset, the region investigated, and the visual contents used
The Spatiotemporal Neural Dynamics of Object Recognition for Natural Images and Line Drawings
Drawings offer a simple and efficient way to communicate meaning. While line drawings capture only coarsely how objects look in reality, we still perceive them as resembling real-world objects. Previous work has shown that this perceived similarity is mirrored by shared neural representations for drawings and natural images, which suggests that similar mechanisms underlie the recognition of both. However, other work has proposed that representations of drawings and natural images become similar only after substantial processing has taken place, suggesting distinct mechanisms. To arbitrate between those alternatives, we measured brain responses resolved in space and time using fMRI and MEG, respectively, while human participants (female and male) viewed images of objects depicted as photographs, line drawings, or sketch-like drawings. Using multivariate decoding, we demonstrate that object category information emerged similarly fast and across overlapping regions in occipital, ventral-temporal, and posterior parietal cortex for all types of depiction, yet with smaller effects at higher levels of visual abstraction. In addition, cross-decoding between depiction types revealed strong generalization of object category information from early processing stages on. Finally, by combining fMRI and MEG data using representational similarity analysis, we found that visual information traversed similar processing stages for all types of depiction, yet with an overall stronger representation for photographs. Together, our results demonstrate broad commonalities in the neural dynamics of object recognition across types of depiction, thus providing clear evidence for shared neural mechanisms underlying recognition of natural object images and abstract drawings
Reliability and generalizability of similarity-based fusion of meg and fmri data in human ventral and dorsal visual streams
To build a representation of what we see, the human brain recruits regions throughout the visual cortex in cascading sequence. Recently, an approach was proposed to evaluate the dynamics of visual perception in high spatiotemporal resolution at the scale of the whole brain. This method combined functional magnetic resonance imaging (fMRI) data with magnetoencephalography (MEG) data using representational similarity analysis and revealed a hierarchical progression from primary visual cortex through the dorsal and ventral streams. To assess the replicability of this method, we here present the results of a visual recognition neuro-imaging fusion experiment and compare them within and across experimental settings. We evaluated the reliability of this method by assessing the consistency of the results under similar test conditions, showing high agreement within participants. We then generalized these results to a separate group of individuals and visual input by comparing them to the fMRI-MEG fusion data of Cichy et al (2016), revealing a highly similar temporal progression recruiting both the dorsal and ventral streams. Together these results are a testament to the reproducibility of the fMRI-MEG fusion approach and allows for the interpretation of these spatiotemporal dynamic in a broader context
Similarity-based fusion of MEG and fMRI discerns early feedforward and feedback processing in the ventral stream
Successful models of vision, such as DNNs and HMAX, are inspired by the human visual system, relying on a hierarchical cascade of feedforward transformations akin to the ventral stream. Despite these advances, the human visual cortex remains unique in complexity, with feedforward and feedback pathways characterized by rapid spatiotemporal dynamics as visual information is transformed into semantic content. Thus, a systematic characterization of the spatiotemporal and representational space of the ventral visual pathway can offer novel insights in the duration and sequencing of cognitive processes, suggesting computational constraints and new architectures for computer vision models.
To discern the feedforward and feedback neural processes underlying human vision, we used MEG/fMRI fusion. We collected MEG data while observers viewed a rapid-serial-visual-presentation of 11 images with an extremely fast speed (17ms/picture or 34ms/picture). Participants performed a two-alternative forced choice task reporting whether the middle image is a face or non-face. fMRI data while observers viewed the same stimuli were also collected.
We used MVPA to pairwise compare all stimuli, creating RDMs separately for MEG and fMRI data. Comparison of time-resolved MEG-RDMs with space-resolved fMRI-RDMs yielded a spatiotemporal description of the ventral stream dynamics. Starting from EVC, brain activation progressed rapidly to IT within approximately 110ms from stimulus onset. The activation cascade reversed back to EVC at around 170ms. This was accompanied by a strengthening of IT activation, leading to categorical representations enhancement.
The presented well-defined spatiotemporal dynamics can be used as constraints for developing new computational neuroscience models with recursive processes, to increase performances in challenging visual conditions