1 research outputs found
From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI
Reconstructing observed images from fMRI brain recordings is challenging.
Unfortunately, acquiring sufficient "labeled" pairs of {Image, fMRI} (i.e.,
images with their corresponding fMRI responses) to span the huge space of
natural images is prohibitive for many reasons. We present a novel approach
which, in addition to the scarce labeled data (training pairs), allows to train
fMRI-to-image reconstruction networks also on "unlabeled" data (i.e., images
without fMRI recording, and fMRI recording without images). The proposed model
utilizes both an Encoder network (image-to-fMRI) and a Decoder network
(fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder
& Decoder-Encoder) allows augmenting the training with both types of unlabeled
data. Importantly, it allows training on the unlabeled test-fMRI data. This
self-supervision adapts the reconstruction network to the new input test-data,
despite its deviation from the statistics of the scarce training data.Comment: *First two authors contributed equally. NeurIPS 201