656 research outputs found

    Graph-Based Decoding Model for Functional Alignment of Unaligned fMRI Data

    Full text link
    Aggregating multi-subject functional magnetic resonance imaging (fMRI) data is indispensable for generating valid and general inferences from patterns distributed across human brains. The disparities in anatomical structures and functional topographies of human brains warrant aligning fMRI data across subjects. However, the existing functional alignment methods cannot handle well various kinds of fMRI datasets today, especially when they are not temporally-aligned, i.e., some of the subjects probably lack the responses to some stimuli, or different subjects might follow different sequences of stimuli. In this paper, a cross-subject graph that depicts the (dis)similarities between samples across subjects is used as a priori for developing a more flexible framework that suits an assortment of fMRI datasets. However, the high dimension of fMRI data and the use of multiple subjects makes the crude framework time-consuming or unpractical. To address this issue, we further regularize the framework, so that a novel feasible kernel-based optimization, which permits nonlinear feature extraction, could be theoretically developed. Specifically, a low-dimension assumption is imposed on each new feature space to avoid overfitting caused by the highspatial-low-temporal resolution of fMRI data. Experimental results on five datasets suggest that the proposed method is not only superior to several state-of-the-art methods on temporally-aligned fMRI data, but also suitable for dealing `with temporally-unaligned fMRI data.Comment: 17 pages, 10 figures, Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI-20

    Neural Encoding and Decoding with Deep Learning for Natural Vision

    Get PDF
    The overarching objective of this work is to bridge neuroscience and artificial intelligence to ultimately build machines that learn, act, and think like humans. In the context of vision, the brain enables humans to readily make sense of the visual world, e.g. recognizing visual objects. Developing human-like machines requires understanding the working principles underlying the human vision. In this dissertation, I ask how the brain encodes and represents dynamic visual information from the outside world, whether brain activity can be directly decoded to reconstruct and categorize what a person is seeing, and whether neuroscience theory can be applied to artificial models to advance computer vision. To address these questions, I used deep neural networks (DNN) to establish encoding and decoding models for describing the relationships between the brain and the visual stimuli. Using the DNN, the encoding models were able to predict the functional magnetic resonance imaging (fMRI) responses throughout the visual cortex given video stimuli; the decoding models were able to reconstruct and categorize the visual stimuli based on fMRI activity. To further advance the DNN model, I have implemented a new bidirectional and recurrent neural network based on the predictive coding theory. As a theory in neuroscience, predictive coding explains the interaction among feedforward, feedback, and recurrent connections. The results showed that this brain-inspired model significantly outperforms feedforward-only DNNs in object recognition. These studies have positive impact on understanding the neural computations under human vision and improving computer vision with the knowledge from neuroscience

    Kernel Methods for Machine Learning with Life Science Applications

    Get PDF

    Characterizing and comparing acoustic representations in convolutional neural networks and the human auditory system

    Full text link
    Le traitement auditif dans le cerveau humain et dans les systèmes informatiques consiste en une cascade de transformations représentationnelles qui extraient et réorganisent les informations pertinentes pour permettre l'exécution des tâches. Cette thèse s'intéresse à la nature des représentations acoustiques et aux principes de conception et d'apprentissage qui soutiennent leur développement. Les objectifs scientifiques sont de caractériser et de comparer les représentations auditives dans les réseaux de neurones convolutionnels profonds (CNN) et la voie auditive humaine. Ce travail soulève plusieurs questions méta-scientifiques sur la nature du progrès scientifique, qui sont également considérées. L'introduction passe en revue les connaissances actuelles sur la voie auditive des mammifères et présente les concepts pertinents de l'apprentissage profond. Le premier article soutient que les questions philosophiques les plus pressantes à l'intersection de l'intelligence artificielle et biologique concernent finalement la définition des phénomènes à expliquer et ce qui constitue des explications valables de tels phénomènes. Je surligne les théories pertinentes de l'explication scientifique que j’espére fourniront un échafaudage pour de futures discussions. L'article 2 teste un modèle populaire de cortex auditif basé sur des modulations spectro-temporelles. Nous constatons qu'un modèle linéaire entraîné uniquement sur les réponses BOLD aux ondulations dynamiques simples (contenant seulement une fréquence fondamentale, un taux de modulation temporelle et une échelle spectrale) peut se généraliser pour prédire les réponses aux mélanges de deux ondulations dynamiques. Le troisième article caractérise la spécificité linguistique des couches CNN et explore l'effet de l'entraînement figé et des poids aléatoires. Nous avons observé trois régions distinctes de transférabilité: (1) les deux premières couches étaient entièrement transférables, (2) les couches 2 à 8 étaient également hautement transférables, mais nous avons trouvé évidence de spécificité de la langue, (3) les couches suivantes entièrement connectées étaient plus spécifiques à la langue mais pouvaient être adaptées sur la langue cible. Dans l'article 4, nous utilisons l'analyse de similarité pour constater que la performance supérieure de l'entraînement figé obtenues à l'article 3 peuvent être attribuées aux différences de représentation dans l'avant-dernière couche: la deuxième couche entièrement connectée. Nous analysons également les réseaux aléatoires de l'article 3, dont nous concluons que la forme représentationnelle est doublement contrainte par l'architecture et la forme de l'entrée et de la cible. Pour tester si les CNN acoustiques apprennent une hiérarchie de représentation similaire à celle du système auditif humain, le cinquième article compare l'activité des réseaux «freeze trained» de l'article 3 à l'activité IRMf 7T dans l'ensemble du système auditif humain. Nous ne trouvons aucune évidence d'une hiérarchie de représentation partagée et constatons plutôt que tous nos régions auditifs étaient les plus similaires à la première couche entièrement connectée. Enfin, le chapitre de discussion passe en revue les mérites et les limites d'une approche d'apprentissage profond aux neurosciences dans un cadre de comparaison de modèles. Ensemble, ces travaux contribuent à l'entreprise naissante de modélisation du système auditif avec des réseaux de neurones et constituent un petit pas vers une science unifiée de l'intelligence qui étudie les phénomènes qui se manifestent dans l'intelligence biologique et artificielle.Auditory processing in the human brain and in contemporary machine hearing systems consists of a cascade of representational transformations that extract and reorganize relevant information to enable task performance. This thesis is concerned with the nature of acoustic representations and the network design and learning principles that support their development. The primary scientific goals are to characterize and compare auditory representations in deep convolutional neural networks (CNNs) and the human auditory pathway. This work prompts several meta-scientific questions about the nature of scientific progress, which are also considered. The introduction reviews what is currently known about the mammalian auditory pathway and introduces the relevant concepts in deep learning.The first article argues that the most pressing philosophical questions at the intersection of artificial and biological intelligence are ultimately concerned with defining the phenomena to be explained and with what constitute valid explanations of such phenomena. I highlight relevant theories of scientific explanation which we hope will provide scaffolding for future discussion. Article 2 tests a popular model of auditory cortex based on frequency-specific spectrotemporal modulations. We find that a linear model trained only on BOLD responses to simple dynamic ripples (containing only one fundamental frequency, temporal modulation rate, and spectral scale) can generalize to predict responses to mixtures of two dynamic ripples. Both the third and fourth article investigate how CNN representations are affected by various aspects of training. The third article characterizes the language specificity of CNN layers and explores the effect of freeze training and random weights. We observed three distinct regions of transferability: (1) the first two layers were entirely transferable between languages, (2) layers 2--8 were also highly transferable but we found some evidence of language specificity, (3) the subsequent fully connected layers were more language specific but could be successfully finetuned to the target language. In Article 4, we use similarity analysis to find that the superior performance of freeze training achieved in Article 3 can be largely attributed to representational differences in the penultimate layer: the second fully connected layer. We also analyze the random networks from Article 3, from which we conclude that representational form is doubly constrained by architecture and the form of the input and target. To test whether acoustic CNNs learn a similar representational hierarchy as that of the human auditory system, the fifth article presents a similarity analysis to compare the activity of the freeze trained networks from Article 3 to 7T fMRI activity throughout the human auditory system. We find no evidence of a shared representational hierarchy and instead find that all of our auditory regions were most similar to the first fully connected layer. Finally, the discussion chapter reviews the merits and limitations of a deep learning approach to neuroscience in a model comparison framework. Together, these works contribute to the nascent enterprise of modeling the auditory system with neural networks and constitute a small step towards a unified science of intelligence that studies the phenomena that are exhibited in both biological and artificial intelligence

    Inimaju arvutuslikke protsesside mõistmine masinõpe mudelite tõlgendamise kaudu. Andmepõhine lähenemine arvutuslikku neuroteadusesse

    Get PDF
    Modelleerimine on inimkonna põline viis keerulistest nähtustest arusaamiseks. Planeetide liikumise mudel, gravitatsiooni mudel ja osakestefüüsika standardmudel on näited selle lähenemise edukusest. Neuroteaduses on olemas kaks viisi mudelite loomiseks: traditsiooniline hüpoteesipõhine lähenemine, mille puhul kõigepealt mudel sõnastatakse ja alles siis valideeritakse andmete peal; ja uuem andmepõhine lähenemine, mis toetub masinõpele, et sõnastada mudeleid automaatselt. Hüpoteesipõhine viis annab täieliku mõistmise sellest, kuidas mudel töötab, aga nõuab aega, kuna iga hüpotees peab olema sõnastatud ja valideeritud käsitsi. Andmepõhine lähenemine toetub ainult andmetele ja arvutuslikele ressurssidele mudelite otsimisel, aga ei seleta kuidas täpselt mudel jõuab oma tulemusteni. Me väidame, et neuroandmestike suur hulk ja nende mahu kiire kasv nõuab andmepõhise lähenemise laiemat kasutuselevõttu neuroteaduses, nihkes uurija rolli mudelite tööprintsiipide tõlgendamisele. Doktoritöö koosneb kolmest näitest neuroteaduse teadmisi avastamisest masinõppe tõlgendamismeetodeid kasutades. Esimeses uuringus tõlgendatava mudeli abiga me kirjeldame millised ajas muutuvad sageduskomponendid iseloomustavad inimese ajusignaali visuaalsete objektide tuvastamise ülesande puhul. Teises uuringus võrdleme omavahel signaale inimese aju ventraalses piirkonnas ja konvolutsiooniliste tehisnärvivõrkude aktivatsioone erinevates kihtides. Säärane võrdlus võimaldas meil kinnitada hüpoteesi, et mõlemad süsteemid kasutavad hierarhilist struktuuri. Viimane näide kasutab topoloogiat säilitavat mõõtmelisuse vähendamise ja visualiseerimise meetodit, et näha, millised ajusignaalid ja mõtteseisundid on üksteisele sarnased. Viimased tulemused masinõppes ja tehisintellektis näitasid et mõned mehhanismid meie ajus on sarnased mehhanismidega, milleni jõuavad õppimise käigus masinõppe algoritmid. Oma tööga me rõhutame masinõppe mudelite tõlgendamise tähtsust selliste mehhanismide avastamiseks.Building a model of a complex phenomenon is an ancient way of gaining knowledge and understanding of the reality around us. Models of planetary motion, gravity, particle physics are examples of this approach. In neuroscience, there are two ways of coming up with explanations of reality: a traditional hypothesis-driven approach, where a model is first formulated and then tested using the data, and a more recent data-driven approach, that relies on machine learning to generate models automatically. Hypothesis-driven approach provides full understanding of the model, but is time-consuming as each model has to be conceived and tested manually. Data-driven approach requires only the data and computational resources to sift through potential models, saving time, but leaving the resulting model itself to be a black box. Given the growing amount of neural data, we argue in favor of a more widespread adoption of the data-driven approach, reallocating part of the human effort from manual modeling. The thesis is based on three examples of how interpretation of machine-learned models leads to neuroscientific insights on three different levels of neural organization. Our first interpretable model is used to characterize neural dynamics of localized neural activity during the task of visual perceptual categorization. Next, we compare the activity of human visual system with the activity of a convolutional neural network, revealing explanations about the functional organization of human visual cortex. Lastly, we use dimensionality reduction and visualization techniques to understand relative organization of mental concepts within a subject's mental state space and apply it in the context of brain-computer interfaces. Recent results in neuroscience and AI show similarities between the mechanisms of both systems. This fact endorses the relevance of our approach: interpreting the mechanisms employed by machine learning models can shed light on the mechanisms employed by our brainhttps://www.ester.ee/record=b536057

    Generative AI for brain image computing and brain network computing: a review

    Get PDF
    Recent years have witnessed a significant advancement in brain imaging techniques that offer a non-invasive approach to mapping the structure and function of the brain. Concurrently, generative artificial intelligence (AI) has experienced substantial growth, involving using existing data to create new content with a similar underlying pattern to real-world data. The integration of these two domains, generative AI in neuroimaging, presents a promising avenue for exploring various fields of brain imaging and brain network computing, particularly in the areas of extracting spatiotemporal brain features and reconstructing the topological connectivity of brain networks. Therefore, this study reviewed the advanced models, tasks, challenges, and prospects of brain imaging and brain network computing techniques and intends to provide a comprehensive picture of current generative AI techniques in brain imaging. This review is focused on novel methodological approaches and applications of related new methods. It discussed fundamental theories and algorithms of four classic generative models and provided a systematic survey and categorization of tasks, including co-registration, super-resolution, enhancement, classification, segmentation, cross-modality, brain network analysis, and brain decoding. This paper also highlighted the challenges and future directions of the latest work with the expectation that future research can be beneficial

    Understanding Stroke in the Connected Human Brain

    Get PDF
    Although structural damage from stroke is focal, remote dysfunction can occur in regions of the brain distant from the area of damage. Lesions in both gray and white matter can disrupt the flow of information in areas connected to or by the area of infarct. This is because the brain is not an assortment of specialized parts but an assembly of distributed networks that interact to support cognitive function. Functional connectivity analyses using resting functional magnetic resonance imaging (fMRI) have shown us that the cortex is organized into distributed brain networks. The primary goal of this work is to characterize the effects of stroke on distributed brain systems and to use this information to better understand neural correlates of deficit and recovery following stroke. We measured resting functional connectivity, lesion topography, and behavior in multiple domains (attention, visual memory, verbal memory, language, motor, and visual) in a cohort of 132 stroke patients. Patients were followed longitudinally with full behavioral and imaging batteries acquired at 2 weeks, 3 months, and 1 year post-stroke. Thirty age- and demographic- matched controls were scanned twice at an interval of three months. In chapter 1, we explore a central question motivating this work: how is behavior represented in the brain? We review progressing prospective – from basic functional localization to newer theories connecting inter-related brain networks to cognitive operations. In so doing, we attempt to build a foundation that motivates the hypotheses and experimental approaches explored in this work. Chapters 2 and 3 serve primarily to validate approaches and considerations for using resting fMRI to measure functional connectivity in stroke patients. In chapter 2, we investigate hemodynamic lags after stroke. ‘Hemodynamic lag’ is a local delay in the blood oxygen level dependent (BOLD) response to neural activity, measured using cross-correlation of local fMRI signal with some reference brain signal. This work tests assumptions of the BOLD response to neural activity after stroke, but also provides novel and clinically relevant insight into perilesional disruption to hemodynamics. Significant lags are observed in 30% of stroke patients sub-acutely and 10% of patients at one-year. Hemodynamic lag corresponds to gross aberrancy in functional connectivity measures, performance deficits and local and global perfusion deficits. Yet, relationships between functional connectivity and behavior reviewed in chapter 1 persist after hemodynamic delays is corrected for. Chapter 3 provides a more extended discussion of approaches and considerations for using resting fMRI to measure functional connectivity in stroke patients. Like chapter 1, the goal is to motivate experimental approaches taken in later chapters. But here, more technical challenges relating to brain co-registration, neurovascular coupling, and clinical population selection are considered. In chapter 4, we uncover the relationships between local damage, network wide functional disconnection, and neurological deficit. We find that visual memory and verbal memory are better predicted by connectivity, whereas visual and motor deficits are better predicted by lesion topography. Attention and language deficits are well predicted by both. We identify a general pattern of physiological network dysfunction consisting of decrease of inter-hemispheric integration and decrease in intra-hemispheric segregation, which strongly related to behavioral impairment in multiple domains. In chapter 5, we explore a case study of abulia – severe apathy. This work ties together principles of local damage, network disruption, and network-related deficit and demonstrates how they can be useful in understanding and developing targeted treatments (such as transcranial magnetic stimulation) for individual stroke patients. In chapter 6, we explore longitudinal changes in functional connectivity that parallel recovery. We find that the topology and boundaries of cortical regions remains unchanged across recovery, empirically validating our parcel-wise connectivity approach. In contrast, we find that the modularity of brain systems i.e. the degree of integration within and segregation between networks, is significantly reduced after a stroke, but partially recovered over time. Importantly, the return of modular network structure parallels recovery of language and attention, but not motor function. This work establishes the importance of normalization of large-scale modular brain systems in stroke recovery. In chapter 7, we discuss some fundamental revisions of past lesion-deficit frameworks necessitated by recent findings. Firstly, anatomical priors of structural and functional connections are needed to explain why certain lesions across distant locations should share behavioral consequences. Secondly, functional priors of connectomics are needed to explain how local injury can produce widespread disruption to brain connectivity and behavior that have been observed
    corecore