310 research outputs found
Constraint-free Natural Image Reconstruction from fMRI Signals Based on Convolutional Neural Network
In recent years, research on decoding brain activity based on functional
magnetic resonance imaging (fMRI) has made remarkable achievements. However,
constraint-free natural image reconstruction from brain activity is still a
challenge. The existing methods simplified the problem by using semantic prior
information or just reconstructing simple images such as letters and digitals.
Without semantic prior information, we present a novel method to reconstruct
nature images from fMRI signals of human visual cortex based on the computation
model of convolutional neural network (CNN). Firstly, we extracted the units
output of viewed natural images in each layer of a pre-trained CNN as CNN
features. Secondly, we transformed image reconstruction from fMRI signals into
the problem of CNN feature visualizations by training a sparse linear
regression to map from the fMRI patterns to CNN features. By iteratively
optimization to find the matched image, whose CNN unit features become most
similar to those predicted from the brain activity, we finally achieved the
promising results for the challenging constraint-free natural image
reconstruction. As there was no use of semantic prior information of the
stimuli when training decoding model, any category of images (not constraint by
the training set) could be reconstructed theoretically. We found that the
reconstructed images resembled the natural stimuli, especially in position and
shape. The experimental results suggest that hierarchical visual features can
effectively express the visual perception process of human brain
Characterization of deep neural network features by decodability from human brain activity
Achievements of near human-level performance in object recognition by deep neural networks (DNNs) have triggered a flood of comparative studies between the brain and DNNs. Using a DNN as a proxy for hierarchical visual representations, our recent study found that human brain activity patterns measured by functional magnetic resonance imaging (fMRI) can be decoded (translated) into DNN feature values given the same inputs. However, not all DNN features are equally decoded, indicating a gap between the DNN and human vision. Here, we present a dataset derived from DNN feature decoding analyses, which includes fMRI signals of five human subjects during image viewing, decoded feature values of DNNs (AlexNet and VGG19), and decoding accuracies of individual DNN features with their rankings. The decoding accuracies of individual features were highly correlated between subjects, suggesting the systematic differences between the brain and DNNs. We hope the present dataset will contribute to revealing the gap between the brain and DNNs and provide an opportunity to make use of the decoded features for further applications
From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI
Reconstructing observed images from fMRI brain recordings is challenging.
Unfortunately, acquiring sufficient "labeled" pairs of {Image, fMRI} (i.e.,
images with their corresponding fMRI responses) to span the huge space of
natural images is prohibitive for many reasons. We present a novel approach
which, in addition to the scarce labeled data (training pairs), allows to train
fMRI-to-image reconstruction networks also on "unlabeled" data (i.e., images
without fMRI recording, and fMRI recording without images). The proposed model
utilizes both an Encoder network (image-to-fMRI) and a Decoder network
(fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder
& Decoder-Encoder) allows augmenting the training with both types of unlabeled
data. Importantly, it allows training on the unlabeled test-fMRI data. This
self-supervision adapts the reconstruction network to the new input test-data,
despite its deviation from the statistics of the scarce training data.Comment: *First two authors contributed equally. NeurIPS 201
Brain2Pix: Fully convolutional naturalistic video reconstruction from brain activity
Reconstructing complex and dynamic visual perception from brain activity remains a major challenge in machine learning applications to neuroscience. Here we present a new method for reconstructing naturalistic images and videos from very large single-participant functional magnetic resonance data that leverages the recent success of image-to-image transformation networks. This is achieved by exploiting spatial information obtained from retinotopic mappings across the visual system. More specifically, we first determine what position each voxel in a particular region of interest would represent in the visual field based on its corresponding receptive field location. Then, the 2D image representation of the brain activity on the visual field is passed to a fully convolutional image-to-image network trained to recover the original stimuli using VGG feature loss with an adversarial regularizer. In our experiments, we show that our method offers a significant improvement over existing video reconstruction technique
Neural Networks with Recurrent Generative Feedback
Neural networks are vulnerable to input perturbations such as additive noise and adversarial attacks. In contrast, human perception is much more robust to such perturbations. The Bayesian brain hypothesis states that human brains use an internal generative model to update the posterior beliefs of the sensory input. This mechanism can be interpreted as a form of self-consistency between the maximum a posteriori (MAP) estimation of an internal generative model and the external environment. Inspired by such hypothesis, we enforce self-consistency in neural networks by incorporating generative recurrent feedback. We instantiate this design
on convolutional neural networks (CNNs). The proposed framework, termed Convolutional Neural Networks with Feedback (CNN-F), introduces a generative feedback with latent variables to existing CNN architectures, where consistent predictions are made through alternating MAP inference under a Bayesian framework. In the experiments, CNN-F shows considerably improved adversarial robustness over conventional feedforward CNNs on standard benchmarks
A deep-dream virtual reality platform for studying altered perceptual phenomenology
Altered states of consciousness, such as psychotic or pharmacologically-induced hallucinations, provide a unique opportunity to examine the mechanisms underlying conscious perception. However, the phenomenological properties of these states are difficult to isolate experimentally from other, more general physiological and cognitive 36 effects of psychoactive substances or psychopathological conditions. Thus, simulating phenomenological aspects of altered states in the absence of these other more general effects provides an important experimental tool for consciousness science and psychiatry. Here we describe such a tool, which we call the Hallucination Machine. It comprises a novel combination of two powerful technologies: deep convolutional neural networks (DCNNs) and panoramic videos of natural scenes, viewed immersively through a head-mounted display (panoramic VR). By doing this, we are able to simulate visual hallucinatory experiences in a biologically plausible and ecologically valid way. Two experiments illustrate potential applications of the Hallucination Machine. First, we show that the system induces visual phenomenology qualitatively similar to classical psychedelics. In a second experiment, we find that simulated hallucinations do not evoke the temporal distortion commonly associated with altered states. Overall, the Hallucination Machine offers a valuable new technique for simulating altered phenomenology without directly altering the underlying neurophysiology
Neural Networks with Recurrent Generative Feedback
Neural networks are vulnerable to input perturbations such as additive noise
and adversarial attacks. In contrast, human perception is much more robust to
such perturbations. The Bayesian brain hypothesis states that human brains use
an internal generative model to update the posterior beliefs of the sensory
input. This mechanism can be interpreted as a form of self-consistency between
the maximum a posteriori (MAP) estimation of an internal generative model and
the external environment. Inspired by such hypothesis, we enforce
self-consistency in neural networks by incorporating generative recurrent
feedback. We instantiate this design on convolutional neural networks (CNNs).
The proposed framework, termed Convolutional Neural Networks with Feedback
(CNN-F), introduces a generative feedback with latent variables to existing CNN
architectures, where consistent predictions are made through alternating MAP
inference under a Bayesian framework. In the experiments, CNN-F shows
considerably improved adversarial robustness over conventional feedforward CNNs
on standard benchmarks.Comment: NeurIPS 202
- …