Reading the mind's eye: Decoding category information during mental imagery

Category information for visually presented objects can be read out from multi-voxel patterns of fMRI activity in ventral–temporal cortex. What is the nature and reliability of these patterns in the absence of any bottom–up visual input, for example, during visual imagery? Here, we first ask how well category information can be decoded for imagined objects and then compare the representations evoked during imagery and actual viewing. In an fMRI study, four object categories (food, tools, faces, buildings) were either visually presented to subjects, or imagined by them. Using pattern classification techniques, we could reliably decode category information (including for non-special categories, i.e., food and tools) from ventral–temporal cortex in both conditions, but only during actual viewing from retinotopic areas. Interestingly, in temporal cortex when the classifier was trained on the viewed condition and tested on the imagery condition, or vice versa, classification performance was comparable to within the imagery condition. The above results held even when we did not use information in the specialized category-selective areas. Thus, the patterns of representation during imagery and actual viewing are in fact surprisingly similar to each other. Consistent with this observation, the maps of “diagnostic voxels” (i.e., the classifier weights) for the perception and imagery classifiers were more similar in ventral–temporal cortex than in retinotopic cortex. These results suggest that in the absence of any bottom–up input, cortical back projections can selectively re-activate specific patterns of neural activity

N1 responses to images of hands in occipito-temporal event-related potentials

Hands, much like faces, convey social information, instructions and intentions to an observer. While the neural processes of face perception have been widely studied, it was only recently that fMRI identified occipito-temporal areas sensitive to static images of hands as body parts. To complement these studies with fine-grained timing information, we measured event-related EEG potentials (ERPs) from 33 subjects who were presented with static images of hands versus faces, whole bodies, and inanimate objects as controls. Already at N1 latency, ~ 170 ms, hand-related ERP patterns were manifest in two results: (1) significant differences in amplitudes for images of hands versus bodies in occipito-temporal N1 responses; (2) left lateralization of responses to images of hands, and also of the difference waveforms (hands minus bodies), quantifying hand-related responses. In line with fMRI studies of hand-sensitive areas distinct from extrastriate body area (EBA), the current findings provide electrophysiological evidence for hand-sensitive brain activation, occurring at a similarly early latency as N1 responses to faces

Holistic Face Categorization in Higher Order Visual Areas of the Normal and Prosopagnosic Brain: Toward a Non-Hierarchical View of Face Perception

How a visual stimulus is initially categorized as a face in a network of human brain areas remains largely unclear. Hierarchical neuro-computational models of face perception assume that the visual stimulus is first decomposed in local parts in lower order visual areas. These parts would then be combined into a global representation in higher order face-sensitive areas of the occipito-temporal cortex. Here we tested this view in fMRI with visual stimuli that are categorized as faces based on their global configuration rather than their local parts (two-tones Mooney figures and Arcimboldo's facelike paintings). Compared to the same inverted visual stimuli that are not categorized as faces, these stimuli activated the right middle fusiform gyrus (“Fusiform face area”) and superior temporal sulcus (pSTS), with no significant activation in the posteriorly located inferior occipital gyrus (i.e., no “occipital face area”). This observation is strengthened by behavioral and neural evidence for normal face categorization of these stimuli in a brain-damaged prosopagnosic patient whose intact right middle fusiform gyrus and superior temporal sulcus are devoid of any potential face-sensitive inputs from the lesioned right inferior occipital cortex. Together, these observations indicate that face-preferential activation may emerge in higher order visual areas of the right hemisphere without any face-preferential inputs from lower order visual areas, supporting a non-hierarchical view of face perception in the visual cortex

The brain as image processor and generator:towards function-restoring brain-computer-interfaces

As neuroscientists are slowly unraveling the mysteries of the brain, neurotechnology like brain-computer-interfaces (BCIs) might become a new standard for medical applications in those with brain injuries. BCIs allow for direct communication between the brain and a device, and could potentially restore links that are broken due to brain damage. In addition, a better understanding of the human mind and its mechanisms could greatly boost the success of these devices. This dissertation features (high-field) functional magnetic resonance imaging (fMRI) to study human cognitive functioning, as fMRI allows for studying the brain of living humans in great spatial detail. Firstly, the dissertation describes how well brain regions that are important for visual perception can be located between individuals. Some of these regions are in part responsible for recognizing objects like faces, bodies, places and motion. Secondly, differences in functional organization of the brain were explored between individuals by simulating the placement of a visual cortical prosthesis. Such a prosthesis can bypass the (broken) connections between the eye and brain in blind people, and potentially restore a rudimentary form of vision. Finally, new techniques were presented that show that visual perception and mental imagery are closely related, and allow for reading letter shapes directly from the mind. Together, this dissertation adds new foundations for the development of neurotechnological applications

Animacy and real-world size shape object representations in the human medial temporal lobes

Identifying what an object is, and whether an object has been encountered before, is a crucial aspect of human behavior. Despite this importance, we do not yet have a complete understanding of the neural basis of these abilities. Investigations into the neural organization of human object representations have revealed category specific organization in the ventral visual stream in perceptual tasks. Interestingly, these categories fall within broader domains of organization, with reported distinctions between animate, inanimate large, and inanimate small objects. While there is some evidence for category specific effects in the medial temporal lobe (MTL), in particular in perirhinal and parahippocampal cortex, it is currently unclear whether domain level organization is also present across these structures. To this end, we used fMRI with a continuous recognition memory task. Stimuli were images of objects from several different categories, which were either animate or inanimate, or large or small within the inanimate domain. We employed representational similarity analysis (RSA) to test the hypothesis that object-evoked responses in MTL structures during recognition-memory judgments also show evidence for domain-level organization along both dimensions. Our data support this hypothesis. Specifically, object representations were shaped by either animacy, real-world size, or both, in perirhinal and parahippocampal cortex, and the hippocampus. While sensitivity to these dimensions differed across structures when probed individually, hinting at interesting links to functional differentiation, similarities in organization across MTL structures were more prominent overall. These results argue for continuity in the organization of object representations in the ventral visual stream and the MTL

Birds of a Feather Flock Together: Experience-Driven Formation of Visual Object Categories in Human Ventral Temporal Cortex

The present functional magnetic resonance imaging study provides direct evidence on visual object-category formation in the human brain. Although brain imaging has demonstrated object-category specific representations in the occipitotemporal cortex, the crucial question of how the brain acquires this knowledge has remained unresolved. We designed a stimulus set consisting of six highly similar bird types that can hardly be distinguished without training. All bird types were morphed with one another to create different exemplars of each category. After visual training, fMRI showed that responses in the right fusiform gyrus were larger for bird types for which a discrete category-boundary was established as compared with not-trained bird types. Importantly, compared with not-trained bird types, right fusiform responses were smaller for visually similar birds to which subjects were exposed during training but for which no category-boundary was learned. These data provide evidence for experience-induced shaping of occipitotemporal responses that are involved in category learning in the human brain

Remote Effects of OFA Disruption on the Face Perception Network Revealed by Consecutive TMS-FMRI

The face perception system is comprised of a network of connected regions including the middle fusiform gyrus (“fusiform face area” or FFA), the inferior occipital gyrus (“occipital face area” or OFA), and the posterior part of the superior temporal sulcus. These regions are typically active bilaterally but may show right hemisphere dominance. The functional magnetic resonance imaging (fMRI) response of the right FFA is normally attenuated for face stimuli of the same compared to different identities, called fMR-adaptation. The recovery in fMRI signal, or release from fMR-adaptation, for faces of different identities indicates that the neural population comprising the FFA is involved in coding face identity. Patients with prosopagnosia who are unable to visually recognize faces and who show right OFA damage, nonetheless show face-selective activation in the right FFA (Rossion et al., 2003; Steeves et al., 2006). However, the sensitivity to face identity is abnormal in the right FFA and does not show the typical release from adaptation for different face identities (Steeves et al., 2009). This indicates that in these patients the FFA is not differentiating face identity and suggests that an intact right OFA is integral for face identity coding. We used offline repetitive transcranial magnetic stimulation (TMS) to temporarily disrupt processing in the right OFA in healthy subjects. We then immediately performed fMRI to measure changes in blood oxygenation level dependent (BOLD) signal across the face network using a face fMR-adaptation paradigm. We hypothesized that TMS to the right OFA would induce abnormal face identity coding in the right FFA, reflected by a decreased adaptation response. Indeed, activation for different but not same identity faces in the right FFA decreased after TMS was applied to the right OFA compared to sham TMS and TMS to a control site, the nearby object-selective right lateral occipital area (LO). Our findings indicate that TMS to the OFA selectively disrupts face but not butterfly identity coding in both the OFA and FFA. Congruent with mounting evidence from both patients and healthy subjects, here we causally demonstrate the importance of the often-overlooked OFA for normal face identity coding in the FFA