5 research outputs found
From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI
Reconstructing observed images from fMRI brain recordings is challenging.
Unfortunately, acquiring sufficient "labeled" pairs of {Image, fMRI} (i.e.,
images with their corresponding fMRI responses) to span the huge space of
natural images is prohibitive for many reasons. We present a novel approach
which, in addition to the scarce labeled data (training pairs), allows to train
fMRI-to-image reconstruction networks also on "unlabeled" data (i.e., images
without fMRI recording, and fMRI recording without images). The proposed model
utilizes both an Encoder network (image-to-fMRI) and a Decoder network
(fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder
& Decoder-Encoder) allows augmenting the training with both types of unlabeled
data. Importantly, it allows training on the unlabeled test-fMRI data. This
self-supervision adapts the reconstruction network to the new input test-data,
despite its deviation from the statistics of the scarce training data.Comment: *First two authors contributed equally. NeurIPS 201
Optimizing deep video representation to match brain activity
International audienceThe comparison of observed brain activity with the statistics generated by artificial intelligence systems is useful to probe brain functional organization under ecological conditions. Here we study fMRI activity in ten subjects watching color natural movies and compute deep representations of these movies with an architecture that relies on optical flow and image content. The association of activity in visual areas with the different layers of the deep architecture displays complexity-related contrasts across visual areas and reveals a striking foveal/peripheral dichotomy