Graph-Based Decoding Model for Functional Alignment of Unaligned fMRI Data

Aggregating multi-subject functional magnetic resonance imaging (fMRI) data is indispensable for generating valid and general inferences from patterns distributed across human brains. The disparities in anatomical structures and functional topographies of human brains warrant aligning fMRI data across subjects. However, the existing functional alignment methods cannot handle well various kinds of fMRI datasets today, especially when they are not temporally-aligned, i.e., some of the subjects probably lack the responses to some stimuli, or different subjects might follow different sequences of stimuli. In this paper, a cross-subject graph that depicts the (dis)similarities between samples across subjects is used as a priori for developing a more flexible framework that suits an assortment of fMRI datasets. However, the high dimension of fMRI data and the use of multiple subjects makes the crude framework time-consuming or unpractical. To address this issue, we further regularize the framework, so that a novel feasible kernel-based optimization, which permits nonlinear feature extraction, could be theoretically developed. Specifically, a low-dimension assumption is imposed on each new feature space to avoid overfitting caused by the highspatial-low-temporal resolution of fMRI data. Experimental results on five datasets suggest that the proposed method is not only superior to several state-of-the-art methods on temporally-aligned fMRI data, but also suitable for dealing `with temporally-unaligned fMRI data.Comment: 17 pages, 10 figures, Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI-20

Building a Science of Individual Differences from fMRI

To date, fMRI research has been concerned primarily with evincing generic principles of brain function through averaging data from multiple subjects. Given rapid developments in both hardware and analysis tools, the field is now poised to study fMRI-derived measures in individual subjects, and to relate these to psychological traits or genetic variations. We discuss issues of validity, reliability and statistical assessment that arise when the focus shifts to individual subjects and that are applicable also to other imaging modalities. We emphasize that individual assessment of neural function with fMRI presents specific challenges and necessitates careful consideration of anatomical and vascular between-subject variability as well as sources of within-subject variability

Application of fMRI for action representation: decoding, aligning and modulating

Functional magnetic resonance imaging (fMRI) is an important tool for understanding neural mechanisms underlying human brain function. Understanding how the human brain responds to stimuli and how different cortical regions represent the information, and if these representational spaces are shared across brains and critical for our understanding of how the brain works. Recently, multivariate pattern analysis (MVPA) has a growing importance to predict mental states from fMRI data and to detect the coarse and fine scale neural responses. However, a major limitation of MVPA is the difficulty of aligning features across brains due to high variability in subjects’ responses and hence MVPA has been generally used as a subject specific analysis. Hyperalignment, solved this problem of feature alignment across brains by mapping neural responses into a common model to facilitate between subject classifications. Another technique of growing importance in understanding brain function is real-time fMRI Neurofeedback, which can be used to enable individuals to alter their brain activity. It facilitates people’s ability to learn control of their cognitive processes like motor control and pain by learning to modulate their brain activation in targeted regions. The aim of this PhD research is to decode and to align the motor representations of multi-joint arm actions based on different modalities of motor simulation, for instance Motor Imagery (MI) and Action Observation (AO) using functional Magnetic Resonance Imaging (fMRI) and to explore the feasibility of using a real-time fMRI neurofeedback to alter these action representations. The first experimental study of this thesis was performed on able-bodied participants to align the neural representation of multi-joint arm actions (lift, knock and throw) during MI tasks in the motor cortex using hyperalignment. Results showed that hyperalignment affords a statistically higher between-subject classification (BSC) performance compared to anatomical alignment. Also, hyperalignment is sensitive to the order in which subjects entered the hyperalignment algorithm to create the common model space. These results demonstrate the effectiveness of hyperalignment to align neural responses in motor cortex across subjects to enable BSC of motor imagery. The second study extended the use of hyperalignment to align fronto-parietal motor regions by addressing the problems of localization and cortical parcellation using cortex based alignment. Also, representational similarity analysis (RSA) was applied to investigate the shared neural code between AO+MI and MI of different actions. Results of MVPA revealed that these actions as well as their modalities can be decoded using the subject’s native or the hyperaligned neural responses. Furthermore, the RSA showed that AO+MI and MI representations formed separate clusters but that the representational organization of action types within these clusters was identical. These findings suggest that the neural representations of AO+MI and MI are neither the same nor totally distinct but exhibit a similar structural geometry with respect to different types of action. Results also showed that MI dominates in the AO+MI condition. The third study was performed on phantom limb pain (PLP) patients to explore the feasibility of using real-time fMRI neurofeedback to down-regulate the activity of premotor (PM) and anterior cingulate (ACC) cortices and whether the successful modulation will reduce the pain intensity. Results demonstrated that PLP patients were able to gain control and decrease the ACC and PM activation. Those patients reported decrease in the ongoing level of pain after training, but it was not statistically significant. The fourth study was conducted on healthy participants to study the effectiveness of fMRI neurofeedback on improving motor function by targeting Supplementary Motor Cortex (SMA). Results showed that participants learnt to up-regulate their SMA activation using MI of complex body actions as a mental strategy. In addition, behavioural changes, i.e. shortening of motor reaction time was found in those participants. These results suggest that fMRI neurofeedback can assist participants to develop greater control over motor regions involved in motor-skill learning and it can be translated into an improvement in motor function. In summary, this PhD thesis extends and validates the usefulness of hyperalignment to align the fronto-parietal motor regions and explores its ability to generalise across different levels of motor representation. Furthermore, it sheds light on the dominant role of MI in the AO+MI condition by examining the neural representational similarity of AO+MI and MI tasks. In addition, the fMRI neurofeedback studies in this thesis provide proof-of-principle of using this technology to reduce pain in clinical applications and to enhance motor functions in a healthy population, with the potential for translation into the clinical environment

Building a Science of Individual Differences from fMRI

To date, fMRI research has been concerned primarily with evincing generic principles of brain function through averaging data from multiple subjects. Given rapid developments in both hardware and analysis tools, the field is now poised to study fMRI-derived measures in individual subjects, and to relate these to psychological traits or genetic variations. We discuss issues of validity, reliability and statistical assessment that arise when the focus shifts to individual subjects and that are applicable also to other imaging modalities. We emphasize that individual assessment of neural function with fMRI presents specific challenges and necessitates careful consideration of anatomical and vascular between-subject variability as well as sources of within-subject variability

Neural Encoding and Decoding with Deep Learning for Natural Vision

The overarching objective of this work is to bridge neuroscience and artificial intelligence to ultimately build machines that learn, act, and think like humans. In the context of vision, the brain enables humans to readily make sense of the visual world, e.g. recognizing visual objects. Developing human-like machines requires understanding the working principles underlying the human vision. In this dissertation, I ask how the brain encodes and represents dynamic visual information from the outside world, whether brain activity can be directly decoded to reconstruct and categorize what a person is seeing, and whether neuroscience theory can be applied to artificial models to advance computer vision. To address these questions, I used deep neural networks (DNN) to establish encoding and decoding models for describing the relationships between the brain and the visual stimuli. Using the DNN, the encoding models were able to predict the functional magnetic resonance imaging (fMRI) responses throughout the visual cortex given video stimuli; the decoding models were able to reconstruct and categorize the visual stimuli based on fMRI activity. To further advance the DNN model, I have implemented a new bidirectional and recurrent neural network based on the predictive coding theory. As a theory in neuroscience, predictive coding explains the interaction among feedforward, feedback, and recurrent connections. The results showed that this brain-inspired model significantly outperforms feedforward-only DNNs in object recognition. These studies have positive impact on understanding the neural computations under human vision and improving computer vision with the knowledge from neuroscience