From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI

Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient "labeled" pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition to the scarce labeled data (training pairs), allows to train fMRI-to-image reconstruction networks also on "unlabeled" data (i.e., images without fMRI recording, and fMRI recording without images). The proposed model utilizes both an Encoder network (image-to-fMRI) and a Decoder network (fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder & Decoder-Encoder) allows augmenting the training with both types of unlabeled data. Importantly, it allows training on the unlabeled test-fMRI data. This self-supervision adapts the reconstruction network to the new input test-data, despite its deviation from the statistics of the scarce training data.Comment: *First two authors contributed equally. NeurIPS 201

Spatio-Temporal Brain Mapping of Motion-Onset VEPs Combined with fMRI and Retinotopic Maps

Neuroimaging studies have identified several motion-sensitive visual areas in the human brain, but the time course of their activation cannot be measured with these techniques. In the present study, we combined electrophysiological and neuroimaging methods (including retinotopic brain mapping) to determine the spatio-temporal profile of motion-onset visual evoked potentials for slow and fast motion stimuli and to localize its neural generators. We found that cortical activity initiates in the primary visual area (V1) for slow stimuli, peaking 100 ms after the onset of motion. Subsequently, activity in the mid-temporal motion-sensitive areas, MT+, peaked at 120 ms, followed by peaks in activity in the more dorsal area, V3A, at 160 ms and the lateral occipital complex at 180 ms. Approximately 250 ms after stimulus onset, activity fast motion stimuli was predominant in area V6 along the parieto-occipital sulcus. Finally, at 350 ms (100 ms after the motion offset) brain activity was visible again in area V1. For fast motion stimuli, the spatio-temporal brain pattern was similar, except that the first activity was detected at 70 ms in area MT+. Comparing functional magnetic resonance data for slow vs. fast motion, we found signs of slow-fast motion stimulus topography along the posterior brain in at least three cortical regions (MT+, V3A and LOR)

Agnosic vision is crowded

Visual agnosia is a neuropsychological impairment of visual object recognition. In spite of over a century of research, little is known about the nature of the functional damage underlying this deficit. We propose that the agnosic deficit is visual crowding: The central vision of agnosic patients is crowded, like the peripheral vision of normally sighted observers. To examine our hypothesis, for each patient who took multiple object-recognition tests, we converted each test score into an equivalent eccentricity, i.e. the eccentricity at which our normally sighted “standard” observer performs as poorly as the centrally-viewing patient. Our standard observer took 15 different screening tests for the diagnosis of visual agnosia at various eccentricities in his periphery. In normal peripheral vision, perception of a simple image (e.g. an isolated letter) is limited by acuity, and perception of a complex image (e.g. a face or a word) is limited by crowding. Our crowding hypothesis proposes that each apperceptive agnosia patient is limited by a degree of crowding that consistently corresponds to one equivalent eccentricity, across all complex images. Analyzing the published data of 32 apperceptive agnosia patients and a group of 14 Posterior Cortical Atrophy (PCA) patients, we find that acuity is spared, and each patient’s pattern of object recognition deficits is well characterized by one number, the equivalent eccentricity (for crowding) at which our standard observer’s peripheral vision is like the central vision of the agnosic patient. In other words, each agnosic patient’s equivalent eccentricity is conserved across tasks. Across patients, equivalent eccentricity ranges from 4 to 40 deg. This indicates that the visual impairment of apperceptive agnosia is crowding. In concert with Song, Levi, & Pelli (2014), we report a double dissociation of acuity and crowding: apperceptive agnosia worsens crowding while sparing acuity, and anisometropic amblyopia worsens acuity while sparing crowding

Perceptual integration and attention in human extrastriate cortex

Abstract Visual crowding is a perceptual phenomenon with far-reaching implications in both perceptual (e.g., object recognition and reading) and clinical (e.g., developmental dyslexia and visual agnosia) domains. Here, we combined event-related fMRI measurements and wide-field brain mapping methods to investigate whether the BOLD response evoked by visual crowding is modulated by different attentional conditions. Participants underwent two sessions of psychophysical training outside the scanner, and then fMRI BOLD activity was measured simultaneously in early visual areas (including the visual word form area, VWFA), while they viewed strongly-crowded and weakly-crowded Gabor patches in attended and unattended conditions. We found that crowding increased BOLD activity in a network of areas including V1, V2, V3A, V4/V8, and VWFA. In V4/V8 and VWFA we found an increased activity related to attention. The effect of crowding in V1 was recorded only when attention was fully devoted to the target location. Our results provide evidence that some area beyond V1 might be the likely candidate for the site of crowding, thus supporting the view of visual crowding as a mid-level visual phenomenon

Visual cortex is sensitive to order-disorder phase transition

Initial stages of visual processing are well characterized in terms of band-limited oriented receptive filters. However, brain mechanisms underlying the integration of their outputs are much less understood. In the domain of texture perception, two types of mechanisms have been suggested: (A) first-order statistics and (B) autocorrelation function. In texture perception, considering local symmetry as a statistical property, we can employ the order parameter used in physics to analyze transitions between order and disorder. When the thermodynamic temperature (T) decreases monotonically, the order parameter changes monotonically from zero for disordered systems to one for symmetric systems. Recently, we have synthesized images corresponding to different T's and showed that human observers are sensitive to phase transition. Their sensitivity function is well approximated by an observer based on the order parameter. Here, we investigated the neural correlates of order-disorder perception using functional imaging combined with a phase-encoded paradigm. We hypothesized that BOLD response would depend monotonically on T if first-order statistics are involved. Conversely, the BOLD response would be larger for images around phase transition than for symmetric and disordered images if autocorrelation is involved, since correlations of all lengths are present only in these images. We presented the stimuli in 4 consecutive 16 s blocks: 1) disordered images, 2) images with continuous change of order parameter from disordered to symmetric, 3) symmetric images, 4) images with continuous change of the order parameter from symmetric to disordered. We found that the BOLD response in early visual areas as well as in lateral occipital complex (LOC) was highest for images close to the phase transition, thus supporting the autocorrelation hypothesis and rejecting first-order statistics as an underlying mechanism. These results may partially account for the weak activation of the LOC to both highly ordered and highly disordered textures compared to object shapes

Self-supervised Natural Image Reconstruction and Large-scale Semantic Classification from Brain Activity

Reconstructing natural images and decoding their semantic category from fMRI brain recordings is challenging. Acquiring sufficient pairs of images and their corresponding fMRI responses, which span the huge space of natural images, is prohibitive. We present a novel self-supervised approach that goes well beyond the scarce paired data, for achieving both: (i) state-of-the art fMRI-to-image reconstruction, and (ii) first-ever large-scale semantic classification from fMRI responses. By imposing cycle consistency between a pair of deep neural networks (from image-to-fMRI & from fMRI-to-image), we train our image reconstruction network on a large number of “unpaired” natural images (images without fMRI recordings) from many novel semantic categories. This enables to adapt our reconstruction network to a very rich semantic coverage without requiring any explicit semantic supervision. Specifically, we find that combining our self-supervised training with high-level perceptual losses, gives rise to new reconstruction & classification capabilities. In particular, this perceptual training enables to classify well fMRIs of never-before-seen semantic classes, without requiring any class labels during training. This gives rise to: (i) Unprecedented image-reconstruction from fMRI of never-before-seen images (evaluated by image metrics and human testing), and (ii) Large-scale semantic classification of categories that were never-before-seen during network training. Such large-scale (1000-way) semantic classification from fMRI recordings has never been demonstrated before. Finally, we provide evidence for the biological consistency of our learned model

Adaptive smoothing based on Gaussian processes regression increases the sensitivity and specificity of fMRI data

Temporal and spatial filtering of fMRI data is often used to improve statistical power. However, conventional methods, such as smoothing with fixed‐width Gaussian filters, remove fine‐scale structure in the data, necessitating a tradeoff between sensitivity and specificity. Specifically, smoothing may increase sensitivity (reduce noise and increase statistical power) but at the cost loss of specificity in that fine‐scale structure in neural activity patterns is lost. Here, we propose an alternative smoothing method based on Gaussian processes (GP) regression for single subjects fMRI experiments. This method adapts the level of smoothing on a voxel by voxel basis according to the characteristics of the local neural activity patterns. GP‐based fMRI analysis has been heretofore impractical owing to computational demands. Here, we demonstrate a new implementation of GP that makes it possible to handle the massive data dimensionality of the typical fMRI experiment. We demonstrate how GP can be used as a drop‐in replacement to conventional preprocessing steps for temporal and spatial smoothing in a standard fMRI pipeline. We present simulated and experimental results that show the increased sensitivity and specificity compared to conventional smoothing strategies. Hum Brain Mapp 38:1438–1459, 2017. © 2016 Wiley Periodicals, Inc

Inhibiting anterior insula changes interoceptive accuracy: a combined TMS-fMRI study

The insular cortex (IC) is involved in sensing and interpreting visceral signals, an ability called interoception, the lack of which is considered a transdiagnostic risk factor for psychopathology. It is still unknown whether it is possible to modulate insular activity to change interoception using noninvasive brain stimulation techniques. Transcranial magnetic stimulation, including theta-burst stimulation (TBS), has proven to be an effective method to non-invasively modulate cortical regions' activity, producing facilitatory (iTBS) or inhibitory (cTBS) effects. By combining TBS with fMRI, we hypothesized that iTBS and cTBS would affect IC activity and, consequently, interoception. Thirty-six participants (18 F; Mage: 23.78 ± 3.56 years) volunteered for this study. cTBS and iTBS, over the right anterior insula, and sham stimulation over vertex were administered in a counterbalanced order across participants. After each stimulation, participants performed the heartbeat counting task and were scanned while performing an explicit emotional judgment task. During this task, they saw disgusting (or, as control, neutral) images that have proven to consistently activate the insula. We found preliminary evidence indicating that cTBS is able to change bilateral anterior IC activation. Specifically, cTBS reduced the bilateral activation of the IC during disgusting blocks and reduced participants’ ability to accurately detect their heartbeats compared to sham. Given the growing use of TMS protocols in psychiatry, current results could be used to inform the conduction of clinical trials aimed at actively changing the IC activity, for example, in patients showing alterations in interoception (e.g., anxiety disorders) or increased disgust sensitivity (e.g., obsessive-compulsive disorders)

Interfering with the activity of the insular cortex to modulate interoceptive awareness: a combined TMS/fMRI study

Aims: Insular cortex (IC) is involved in sensing, interpreting and being aware of signals coming from the viscera, an ability called interoceptive awareness (IA), the lack of which is considered a transdiagnostic risk factor for psychopathology1. However, it is still unknown if and to what extent it is possible to modulate IC activity to eventually modify individuals’ IA by using noninvasive brain stimulation techniques. Transcranial magnetic stimulation, including theta-burst stimulation (TBS) protocols, has proven to be an effective method to non-invasively modulate a variety of cortical regions' activity, producing facilitatory (iTBS) or inhibitory (cTBS) effects. It has been recently demonstrated that cTBS delivered over a fronto-temporal region changes participants’ performance at the heartbeat counting task2, a measure of IA. However, it has not been tested yet if this is because TBS changes excitability of the IC. By combining TBS with fMRI, we hypothesized that iTBS and cTBS would affect IC activity and, consequently, interoceptive awareness. Materials and Methods: 22 healthy individuals (13 F/ 9 M; mean age: 23.86 ± 3.91 years) participated in this study. cTBS and iTBS, over the right anterior IC and a sham stimulation over a control brain region were administered in a counterbalanced order across participants. After each stimulation, participants performed an IA task and were scanned throughout a MRI 3T Siemens, while performing an explicit emotional judgment task. During this task, they saw disgusting or neutral images from the dataset DIRTI3 that has proven to consistently activate the insula. The task was composed by 20 blocks, and each block was composed by six images. After each block of images, participants reported on a 5-point Likert scale how much they felt disgusted. Results: We found preliminary evidence indicating that the two TBS sessions changed bilateral IC activation. In particular, the cTBS, reduced bilateral activation of the anterior IC when disgusting images were displayed. Discussion and Conclusions: Given the growing use of TMS protocols in psychiatry, current results could be used to inform the conduction of clinical trials aimed at facilitating or inhibiting the activity of the insular cortex (for example in patients showing impaired interoception) with the ultimate goal to harness scientific advances to select treatment options with the greatest likelihood of success. 1)Craig, A. D. (2009). How do you feel—now? The anterior insula and human awareness. Nature reviews neuroscience, 10(1), 59-70. 2)Pollatos, O., Herbert, B. M., Mai, S., & Kammer, T. (2016). Changes in interoceptive processes following brain stimulation. Philosophical Transactions of the Royal Society B: Biological Sciences, 371(1708) 3)Haberkamp, A., Glombiewski, J. A., Schmidt, F., & Barke, A. (2017). The DIsgust-RelaTed-Images (DIRTI) database: Validation of a novel standardized set of disgust pictures. Behaviour research and therapy, 89, 86-94