Transfer learning of deep neural network representations for fMRI decoding

Background: Deep neural networks have revolutionised machine learning, with unparalleled performance in object classification. However, in brain imaging (e.g., fMRI), the direct application of Convolutional Neural Networks (CNN) to decoding subject states or perception from imaging data seems impractical given the scarcity of available data. New method: In this work we propose a robust method to transfer information from deep learning (DL) features to brain fMRI data with the goal of decoding. By adopting Reduced Rank Regression with Ridge Regularisation we establish a multivariate link between imaging data and the fully connected layer (fc7) of a CNN. We exploit the reconstructed fc7 features by performing an object image classification task on two datasets: one of the largest fMRI databases, taken from different scanners from more than two hundred subjects watching different movie clips, and another with fMRI data taken while watching static images. Results: The fc7 features could be significantly reconstructed from the imaging data, and led to significant decoding performance. Comparison with existing methods: The decoding based on reconstructed fc7 outperformed the decoding based on imaging data alone. Conclusion: In this work we show how to improve fMRI-based decoding benefiting from the mapping between functional data and CNN features. The potential advantage of the proposed method is twofold: the extraction of stimuli representations by means of an automatic procedure (unsupervised) and the embedding of high-dimensional neuroimaging data onto a space designed for visual object discrimination, leading to a more manageable space from dimensionality point of view

Naturalistic language comprehension : a fMRI study on semantics in a narrative context

Semantiikka tutkii kieleen sisältyviä merkityksiä, joita tarvitaan kielen ymmärryksessä. Kuinka aivomme käsittelevät semantiikkaa ja kuinka ymmärrämme erityisesti luonnollisessa muodossa olevaa kieltä, on vielä aivotutkijoille epäselvää. Tässä tutkimuksessa kysyttiin, miten laajemmassa kontekstissa, narratiivissa, olevan kielen ymmärrys ja semanttinen prosessointi heijastuu aivojen aktiivisuuteen. Koehenkilöt kuulivat narratiivin toiminnallisen magneettiresonanssikuvantamisen (fMRI) aikana. Narratiivin semanttinen sisältö mallinnettiin laskennallisesti word2vec algoritmin avulla, ja tätä mallia verrattiin veren happitasosta riippuvaiseen (BOLD) aivosignaaliin ridge regression avulla vokseli kerrallaan. Lähestymistavalla saatiin eristettyä yksityiskohtaisempaa tietoa jatkuvan stimuluksen aivodatasta perustuen kielen semanttiseen sisältöön. Subjektien välinen BOLD-signaalin korrelaatio (ISC) itsessään paljasti molempien aivopuoliskojen osallistuvan kielen ymmärrykseen laajasti. Alueellista päällekkäisyyttä löytyi muiden aivoverkostojen kanssa, jotka vastaavat mm. mentalisaatiosta, muistista ja keskittymiskyvystä, mikä viittaa kielen ymmärryksen vaativan myös muiden kognition osien toimintaa. Ridge regression tulokset viittaavat bilateraalisten pikkuaivojen, superiorisen, keskimmäisen sekä mediaalisen etuaivokuoren poimujen, inferiorisen ja mediaalisen parietaalikuoren sekä visuaalikuoren, sekä oikean temporaalikuoren osallistuvan narratiivin semanttiseen prosessointiin aivoissa. Aiempi semantiikan tutkimus on tuottanut samankaltaisia tuloksia, joten word2vec vaikuttaisi tämän tutkimuksen perusteella mallintavan semantiikkaa riittävän hyvin aivotutkimuksen tarpeisiin. Tutkimuksen perusteella molemmat aivopuoliskot osallistuvat kielen laajemman kontekstin käsittelyyn, ja semantiikka nähdään aktivaationa eri puolilla aivokuorta. Nämä aktiivisuudet ovat mahdollisesti riippuvaisia kielen sisällöstä, mutta miten paljon kielen sisältö vaikuttaa eri aivoalueiden osallistumiseen kielen semanttisessa prosessoinnissa, on vielä avoin tutkimuskysymys.Semantics is a study of meaning in language and basis for language comprehension. How these phenomena are processed in the brain is still unclear especially in naturalistic context. In this study, naturalistic language comprehension, and how semantic processing in a narrative context is reflected in brain activity were investigated. Subjects were measured with functional magnetic resonance imaging (fMRI) while listening to a narrative. The semantic content of the narrative was modelled computationally with word2vec and compared to voxel-wise blood-oxygen-level dependent (BOLD) brain signal time courses using ridge regression. This approach provides a novel way to extract more detailed information from the brain data based on semantic content of the stimulus. Inter-subject correlation (ISC) of voxel-wise BOLD signals alone showed both hemispheres taking part in language comprehension. Areas involved in this task overlapped with networks of mentalisation, memory and attention suggesting comprehension requiring other modalities of cognition for its function. Ridge regression suggested cerebellum, superior, middle and medial frontal, inferior and medial parietal and visual cortices bilaterally and temporal cortex on right hemisphere having a role in semantic processing of the narrative. As similar results have been found in previous research on semantics, word2vec appears to model semantics sufficiently and is an applicable tool in brain research. This study suggests contextual language recruiting brain areas in both hemispheres and semantic processing showing as distributed activity on the cortex. This activity is likely dependent on the content of language, but further studies are required to distinguish how strongly brain activity is affected by different semantic contents

Robust inter-subject audiovisual decoding in functional magnetic resonance imaging using high-dimensional regression

Major methodological advancements have been recently made in the field of neural decoding, which is concerned with the reconstruction of mental content from neuroimaging measures. However, in the absence of a large-scale examination of the validity of the decoding models across subjects and content, the extent to which these models can be generalized is not clear. This study addresses the challenge of producing generalizable decoding models, which allow the reconstruction of perceived audiovisual features from human magnetic resonance imaging (fMRI) data without prior training of the algorithm on the decoded content. We applied an adapted version of kernel ridge regression combined with temporal optimization on data acquired during film viewing (234 runs) to generate standardized brain models for sound loudness, speech presence, perceived motion, face-to-frame ratio, lightness, and color brightness. The prediction accuracies were tested on data collected from different subjects watching other movies mainly in another scanner. Substantial and significant (QFDR<0.05) correlations between the reconstructed and the original descriptors were found for the first three features (loudness, speech, and motion) in all of the 9 test movies (R¯=0.62, R¯ = 0.60, R¯ = 0.60, respectively) with high reproducibility of the predictors across subjects. The face ratio model produced significant correlations in 7 out of 8 movies (R¯=0.56). The lightness and brightness models did not show robustness (R¯=0.23, R¯ = 0). Further analysis of additional data (95 runs) indicated that loudness reconstruction veridicality can consistently reveal relevant group differences in musical experience. The findings point to the validity and generalizability of our loudness, speech, motion, and face ratio models for complex cinematic stimuli (as well as for music in the case of loudness). While future research should further validate these models using controlled stimuli and explore the feasibility of extracting more complex models via this method, the reliability of our results indicates the potential usefulness of the approach and the resulting models in basic scientific and diagnostic contexts

Neural Encoding and Decoding with Deep Learning for Natural Vision

The overarching objective of this work is to bridge neuroscience and artificial intelligence to ultimately build machines that learn, act, and think like humans. In the context of vision, the brain enables humans to readily make sense of the visual world, e.g. recognizing visual objects. Developing human-like machines requires understanding the working principles underlying the human vision. In this dissertation, I ask how the brain encodes and represents dynamic visual information from the outside world, whether brain activity can be directly decoded to reconstruct and categorize what a person is seeing, and whether neuroscience theory can be applied to artificial models to advance computer vision. To address these questions, I used deep neural networks (DNN) to establish encoding and decoding models for describing the relationships between the brain and the visual stimuli. Using the DNN, the encoding models were able to predict the functional magnetic resonance imaging (fMRI) responses throughout the visual cortex given video stimuli; the decoding models were able to reconstruct and categorize the visual stimuli based on fMRI activity. To further advance the DNN model, I have implemented a new bidirectional and recurrent neural network based on the predictive coding theory. As a theory in neuroscience, predictive coding explains the interaction among feedforward, feedback, and recurrent connections. The results showed that this brain-inspired model significantly outperforms feedforward-only DNNs in object recognition. These studies have positive impact on understanding the neural computations under human vision and improving computer vision with the knowledge from neuroscience