Review: Object vision in a structured world

In natural vision, objects appear at typical locations, both with respect to visual space (e.g., an airplane in the upper part of a scene) and other objects (e.g., a lamp above a table). Recent studies have shown that object vision is strongly adapted to such positional regularities. In this review we synthesize these developments, highlighting that adaptations to positional regularities facilitate object detection and recognition, and sharpen the representations of objects in visual cortex. These effects are pervasive across various types of high-level content. We posit that adaptations to real-world structure collectively support optimal usage of limited cortical processing resources. Taking positional regularities into account will thus be essential for understanding efficient object vision in the real world

Modeling biological face recognition with deep convolutional neural networks

Deep convolutional neural networks (DCNNs) have become the state-of-the-art computational models of biological object recognition. Their remarkable success has helped vision science break new ground and recent efforts have started to transfer this achievement to research on biological face recognition. In this regard, face detection can be investigated by comparing face-selective biological neurons and brain areas to artificial neurons and model layers. Similarly, face identification can be examined by comparing in vivo and in silico multidimensional "face spaces". In this review, we summarize the first studies that use DCNNs to model biological face recognition. On the basis of a broad spectrum of behavioral and computational evidence, we conclude that DCNNs are useful models that closely resemble the general hierarchical organization of face recognition in the ventral visual pathway and the core face network. In two exemplary spotlights, we emphasize the unique scientific contributions of these models. First, studies on face detection in DCNNs indicate that elementary face selectivity emerges automatically through feedforward processing even in the absence of visual experience. Second, studies on face identification in DCNNs suggest that identity-specific experience and generative mechanisms facilitate this particular challenge. Taken together, as this novel modeling approach enables close control of predisposition (i.e., architecture) and experience (i.e., training data), it may be suited to inform long-standing debates on the substrates of biological face recognition.Comment: 41 pages, 2 figures, 1 tabl

ECoG high gamma activity reveals distinct cortical representations of lyrics passages, harmonic and timbre-related changes in a rock song

Listening to music moves our minds and moods, stirring interest in its neural underpinnings. A multitude of compositional features drives the appeal of natural music. How such original music, where a composer's opus is not manipulated for experimental purposes, engages a listener's brain has not been studied until recently. Here, we report an in-depth analysis of two electrocorticographic (ECoG) data sets obtained over the left hemisphere in ten patients during presentation of either a rock song or a read-out narrative. First, the time courses of five acoustic features (intensity, presence/absence of vocals with lyrics, spectral centroid, harmonic change, and pulse clarity) were extracted from the audio tracks and found to be correlated with each other to varying degrees. In a second step, we uncovered the specific impact of each musical feature on ECoG high-gamma power (70–170 Hz) by calculating partial correlations to remove the influence of the other four features. In the music condition, the onset and offset of vocal lyrics in ongoing instrumental music was consistently identified within the group as the dominant driver for ECoG high-gamma power changes over temporal auditory areas, while concurrently subject-individual activation spots were identified for sound intensity, timbral, and harmonic features. The distinct cortical activations to vocal speech-related content embedded in instrumental music directly demonstrate that song integrated in instrumental music represents a distinct dimension in complex music. In contrast, in the speech condition, the full sound envelope was reflected in the high gamma response rather than the onset or offset of the vocal lyrics. This demonstrates how the contributions of stimulus features that modulate the brain response differ across the two examples of a full-length natural stimulus, which suggests a context-dependent feature selection in the processing of complex auditory stimuli

Atypical eye contact in autism: Models, mechanisms and development

An atypical pattern of eye contact behaviour is one of the most significant symptoms of Autism Spectrum Disorder (ASD). Recent empirical advances have revealed the developmental, cognitive and neural basis of atypical eye contact behaviour in ASD. We review different models and advance a new ‘fast-track modulator model’. Specifically, we propose that atypical eye contact processing in ASD originates in the lack of influence from a subcortical face and eye contact detection route, which is hypothesized to modulate eye contact processing and guide its emergent specialization during development

An fMRI dataset in response to “The Grand Budapest Hotel”, a socially-rich, naturalistic movie

Naturalistic stimuli evoke strong, consistent, and information-rich patterns of brain activity, and engage large extents of the human brain. They allow researchers to compare highly similar brain responses across subjects, and to study how complex representations are encoded in brain activity. Here, we describe and share a dataset where 25 subjects watched part of the feature film “The Grand Budapest Hotel” by Wes Anderson. The movie has a large cast with many famous actors. Throughout the story, the camera shots highlight faces and expressions, which are fundamental to understand the complex narrative of the movie. This movie was chosen to sample brain activity specifically related to social interactions and face processing. This dataset provides researchers with fMRI data that can be used to explore social cognitive processes and face processing, adding to the existing neuroimaging datasets that sample brain activity with naturalistic movies

Beliefs about the Minds of Others Influence How We Process Sensory Information

Attending where others gaze is one of the most fundamental mechanisms of social cognition. The present study is the first to examine the impact of the attribution of mind to others on gaze-guided attentional orienting and its ERP correlates. Using a paradigm in which attention was guided to a location by the gaze of a centrally presented face, we manipulated participants' beliefs about the gazer: gaze behavior was believed to result either from operations of a mind or from a machine. In Experiment 1, beliefs were manipulated by cue identity (human or robot), while in Experiment 2, cue identity (robot) remained identical across conditions and beliefs were manipulated solely via instruction, which was irrelevant to the task. ERP results and behavior showed that participants' attention was guided by gaze only when gaze was believed to be controlled by a human. Specifically, the P1 was more enhanced for validly, relative to invalidly, cued targets only when participants believed the gaze behavior was the result of a mind, rather than of a machine. This shows that sensory gain control can be influenced by higher-order (task-irrelevant) beliefs about the observed scene. We propose a new interdisciplinary model of social attention, which integrates ideas from cognitive and social neuroscience, as well as philosophy in order to provide a framework for understanding a crucial aspect of how humans' beliefs about the observed scene influence sensory processing

Approaches to Understanding the Function of Intrinsic Activity and its Relationship to Task-evoked Activity in the Human Brain

Traditionally neuroscience research has focused on characterizing the topography and patterns of brain activation evoked by specific cognitive or behavioral tasks to understand human brain functions. This activation-based paradigm treated underlying spontaneous brain activity, a.k.a. intrinsic activity, as noise hence irrelevant to cognitive or behavioral functions. This view, however, has been profoundly modified by the discovery that intrinsic activity is not random, but temporally correlated at rest in widely distributed spatiotemporal patterns, so called resting state networks (RSN). Studies of temporal correlation of spontaneous activity among brain regions, or functional connectivity (FC), have yielded important insights into the network organization of the human brain. However, the underlying fundamental relationship between intrinsic and task-evoked brain activity has remained unclear, becoming an increasingly important topic in neuroscience. An emerging view is that neural activity evoked by a task and the associated behavior is influenced and constrained by intrinsic activity. Additionally, intrinsic activity may be shaped in the course of development or adult life by neural activity evoked by a task through a Hebbian learning process. This thesis aims to reveal correspondences between intrinsic activity and task-evoked activity to better understand the nature and function of intrinsic brain activity. We measured in human visual cortex the blood oxygen level dependent (BOLD) signal with fMRI to analyze the multivoxel activity patterns and FC structures of intrinsic activity, and compare them to those evoked by natural and synthetic visual stimuli. In chapter 1, we review previous evidence of an association between intrinsic and task-evoked activity across studies using different experimental methods. Two experimental strategies from the literature were adapted to our own experiments. First, from anesthetized animal studies of intrinsic activity in visual cortex, we set out to measure macro-scale multi-voxel patterns of spontaneous activity fluctuations as they relate to visually driven patterns of activity (Chapters 2 and 4). Second, from inter-subject correlation studies of visual activity driven by natural stimuli, we measure relationships between intrinsic and evoked activity, specifically in relation to their topographic similarity at the network level (Chapter 5). In Chapter 2 to 4, we establish a multivariate-pattern analysis (MVPA) approach to evaluate patterns of intrinsic and task-evoked activity. The main idea is that patterns of activity induced by behaviorally relevant stimuli over long periods of time would be represented in spontaneous activity patterns within the same areas. To test the idea, in Chapter 2, we compare the overall degree of pattern similarities between resting-state activity patterns, frame-by-frame (framewise), and visual-stimulus evoked activity patterns for natural (face, body, scenes, man-made objects) and synthetic (phase and position scrambled) object images during low-level detection task. We found that the variability, not the mean, of pattern similarity was significantly higher for natural than synthetic stimuli in visual occipital regions that preferred particular stimulus categories. Chapter 3 extends the static categorical pattern similarity measure of Chapter 2 into a temporal correlation measure. We built pattern-based FC matrices for different stimulus categories (e.g. a face specific multivoxel pattern) in regions that preferred particular stimulus categories (e.g. FFA, STG), and showed that the occurrence of a specific categorical pattern generalizes across category specific regions. These pattern-based FCs resemble that of resting-state FC of the same regions supporting that resting state patterns are related to category-specific stimulus-evoked multivoxel activity patterns. In Chapter 4, we repeat the analysis used in Chapter 2 with language stimuli. Language stimuli (alphabetic letters and English words) are interesting as they are learned through intensive training as kids learn to read. Therefore, they represent a non-natural category of stimuli that is, however, highly trained in literate individuals. The visual stimuli used in Chapter 2 to 4 are designed specifically for a laboratory environment that does not correspond to realistic ecological environments. In Chapter 5, to overcome this limitation, we use the more naturalistic visual experience of movie-watching and compare the whole-brain FC network structure of movie-watching and of resting-state. We show the whole-brain FC structure evoked by movie-watching is partly constrained by the resting network structure. In conclusion, our experiments show that the link between intrinsic activity and task-evoked activity is not only limited to inter-regional interactions (as in regular resting-state FC), hence potentially reflecting anatomical connectivity or modulations of excitability between cortical regions, but extends to multivoxel patterns that carry information about specific stimulus categories. This result supports the notion that intrinsic activity constrains task-evoked, not only in terms of topography or activation levels, but also in terms of the information states that are represented in cortex

Modeling Semantic Encoding in a Common Neural Representational Space

Encoding models for mapping voxelwise semantic tuning are typically estimated separately for each individual, limiting their generalizability. In the current report, we develop a method for estimating semantic encoding models that generalize across individuals. Functional MRI was used to measure brain responses while participants freely viewed a naturalistic audiovisual movie. Word embeddings capturing agent-, action-, object-, and scene-related semantic content were assigned to each imaging volume based on an annotation of the film. We constructed both conventional within-subject semantic encoding models and between-subject models where the model was trained on a subset of participants and validated on a left-out participant. Between-subject models were trained using cortical surface-based anatomical normalization or surface-based whole-cortex hyperalignment. We used hyperalignment to project group data into an individual’s unique anatomical space via a common representational space, thus leveraging a larger volume of data for out-of-sample prediction while preserving the individual’s fine-grained functional–anatomical idiosyncrasies. Our findings demonstrate that anatomical normalization degrades the spatial specificity of between-subject encoding models relative to within-subject models. Hyperalignment, on the other hand, recovers the spatial specificity of semantic tuning lost during anatomical normalization, and yields model performance exceeding that of within-subject models

Using child-friendly movie stimuli to study the development of face, place, and object regions from age 3 to 12 years

Scanning young children while they watch short, engaging, commercially‐produced movies has emerged as a promising approach for increasing data retention and quality. Movie stimuli also evoke a richer variety of cognitive processes than traditional experiments, allowing the study of multiple aspects of brain development simultaneously. However, because these stimuli are uncontrolled, it is unclear how effectively distinct profiles of brain activity can be distinguished from the resulting data. Here we develop an approach for identifying multiple distinct subject‐specific Regions of Interest (ssROIs) using fMRI data collected during movie‐viewing. We focused on the test case of higher‐level visual regions selective for faces, scenes, and objects. Adults (N = 13) were scanned while viewing a 5.6‐min child‐friendly movie, as well as a traditional localizer experiment with blocks of faces, scenes, and objects. We found that just 2.7 min of movie data could identify subject‐specific face, scene, and object regions. While successful, movie‐defined ssROIS still showed weaker domain selectivity than traditional ssROIs. Having validated our approach in adults, we then used the same methods on movie data collected from 3 to 12‐year‐old children (N = 122). Movie response timecourses in 3‐year‐old children's face, scene, and object regions were already significantly and specifically predicted by timecourses from the corresponding regions in adults. We also found evidence of continued developmental change, particularly in the face‐selective posterior superior temporal sulcus. Taken together, our results reveal both early maturity and functional change in face, scene, and object regions, and more broadly highlight the promise of short, child‐friendly movies for developmental cognitive neuroscience

Lipreading a naturalistic narrative in a female population : Neural characteristics shared with listening and reading

Publisher Copyright: © 2022 The Authors. Brain and Behavior published by Wiley Periodicals LLC.Introduction: Few of us are skilled lipreaders while most struggle with the task. Neural substrates that enable comprehension of connected natural speech via lipreading are not yet well understood. Methods: We used a data-driven approach to identify brain areas underlying the lipreading of an 8-min narrative with participants whose lipreading skills varied extensively (range 6–100%, mean = 50.7%). The participants also listened to and read the same narrative. The similarity between individual participants’ brain activity during the whole narrative, within and between conditions, was estimated by a voxel-wise comparison of the Blood Oxygenation Level Dependent (BOLD) signal time courses. Results: Inter-subject correlation (ISC) of the time courses revealed that lipreading, listening to, and reading the narrative were largely supported by the same brain areas in the temporal, parietal and frontal cortices, precuneus, and cerebellum. Additionally, listening to and reading connected naturalistic speech particularly activated higher-level linguistic processing in the parietal and frontal cortices more consistently than lipreading, probably paralleling the limited understanding obtained via lip-reading. Importantly, higher lipreading test score and subjective estimate of comprehension of the lipread narrative was associated with activity in the superior and middle temporal cortex. Conclusions: Our new data illustrates that findings from prior studies using well-controlled repetitive speech stimuli and stimulus-driven data analyses are also valid for naturalistic connected speech. Our results might suggest an efficient use of brain areas dealing with phonological processing in skilled lipreaders.Peer reviewe