Early soft and flexible fusion of electroencephalography and functional magnetic resonance imaging via double coupled matrix tensor factorization for multisubject group analysis

Data fusion refers to the joint analysis of multiple datasets that provide different (e.g., complementary) views of the same task. In general, it can extract more information than separate analyses can. Jointly analyzing electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) measurements has been proved to be highly beneficial to the study of the brain function, mainly because these neuroimaging modalities have complementary spatiotemporal resolution: EEG offers good temporal resolution while fMRI is better in its spatial resolution. The EEG–fMRI fusion methods that have been reported so far ignore the underlying multiway nature of the data in at least one of the modalities and/or rely on very strong assumptions concerning the relation of the respective datasets. For example, in multisubject analysis, it is commonly assumed that the hemodynamic response function is a priori known for all subjects and/or the coupling across corresponding modes is assumed to be exact (hard). In this article, these two limitations are overcome by adopting tensor models for both modalities and by following soft and flexible coupling approaches to implement the multimodal fusion. The obtained results are compared against those of parallel independent component analysis and hard coupling alternatives, with both synthetic and real data (epilepsy and visual oddball paradigm). Our results demonstrate the clear advantage of using soft and flexible coupled tensor decompositions in scenarios that do not conform with the hard coupling assumption

Multimodal Data Fusion: An Overview of Methods, Challenges and Prospects

International audienceIn various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or subjects, among others. We use the term "modality" for each such acquisition framework. Due to the rich characteristics of natural phenomena, it is rare that a single modality provides complete knowledge of the phenomenon of interest. The increasing availability of several modalities reporting on the same system introduces new degrees of freedom, which raise questions beyond those related to exploiting each modality separately. As we argue, many of these questions, or "challenges" , are common to multiple domains. This paper deals with two key questions: "why we need data fusion" and "how we perform it". The first question is motivated by numerous examples in science and technology, followed by a mathematical framework that showcases some of the benefits that data fusion provides. In order to address the second question, "diversity" is introduced as a key concept, and a number of data-driven solutions based on matrix and tensor decompositions are discussed, emphasizing how they account for diversity across the datasets. The aim of this paper is to provide the reader, regardless of his or her community of origin, with a taste of the vastness of the field, the prospects and opportunities that it holds

Unraveling Diagnostic Biomarkers of Schizophrenia Through Structure-Revealing Fusion of Multi-Modal Neuroimaging Data

Fusing complementary information from different modalities can lead to the discovery of more accurate diagnostic biomarkers for psychiatric disorders. However, biomarker discovery through data fusion is challenging since it requires extracting interpretable and reproducible patterns from data sets, consisting of shared/unshared patterns and of different orders. For example, multi-channel electroencephalography (EEG) signals from multiple subjects can be represented as a third-order tensor with modes: subject, time, and channel, while functional magnetic resonance imaging (fMRI) data may be in the form of subject by voxel matrices. Traditional data fusion methods rearrange higher-order tensors, such as EEG, as matrices to use matrix factorization-based approaches. In contrast, fusion methods based on coupled matrix and tensor factorizations (CMTF) exploit the potential multi-way structure of higher-order tensors. The CMTF approach has been shown to capture underlying patterns more accurately without imposing strong constraints on the latent neural patterns, i.e., biomarkers. In this paper, EEG, fMRI, and structural MRI (sMRI) data collected during an auditory oddball task (AOD) from a group of subjects consisting of patients with schizophrenia and healthy controls, are arranged as matrices and higher-order tensors coupled along the subject mode, and jointly analyzed using structure-revealing CMTF methods [also known as advanced CMTF (ACMTF)] focusing on unique identification of underlying patterns in the presence of shared/unshared patterns. We demonstrate that joint analysis of the EEG tensor and fMRI matrix using ACMTF reveals significant and biologically meaningful components in terms of differentiating between patients with schizophrenia and healthy controls while also providing spatial patterns with high resolution and improving the clustering performance compared to the analysis of only the EEG tensor. We also show that these patterns are reproducible, and study reproducibility for different model parameters. In comparison to the joint independent component analysis (jICA) data fusion approach, ACMTF provides easier interpretation of EEG data by revealing a single summary map of the topography for each component. Furthermore, fusion of sMRI data with EEG and fMRI through an ACMTF model provides structural patterns; however, we also show that when fusing data sets from multiple modalities, hence of very different nature, preprocessing plays a crucial role

Tensor Regression

Regression analysis is a key area of interest in the field of data analysis and machine learning which is devoted to exploring the dependencies between variables, often using vectors. The emergence of high dimensional data in technologies such as neuroimaging, computer vision, climatology and social networks, has brought challenges to traditional data representation methods. Tensors, as high dimensional extensions of vectors, are considered as natural representations of high dimensional data. In this book, the authors provide a systematic study and analysis of tensor-based regression models and their applications in recent years. It groups and illustrates the existing tensor-based regression methods and covers the basics, core ideas, and theoretical characteristics of most tensor-based regression methods. In addition, readers can learn how to use existing tensor-based regression methods to solve specific regression tasks with multiway data, what datasets can be selected, and what software packages are available to start related work as soon as possible. Tensor Regression is the first thorough overview of the fundamentals, motivations, popular algorithms, strategies for efficient implementation, related applications, available datasets, and software resources for tensor-based regression analysis. It is essential reading for all students, researchers and practitioners of working on high dimensional data.Comment: 187 pages, 32 figures, 10 table

Tensor-based regression models and applications

Tableau d’honneur de la Faculté des études supérieures et postdoctorales, 2017-2018Avec l’avancement des technologies modernes, les tenseurs d’ordre élevé sont assez répandus et abondent dans un large éventail d’applications telles que la neuroscience informatique, la vision par ordinateur, le traitement du signal et ainsi de suite. La principale raison pour laquelle les méthodes de régression classiques ne parviennent pas à traiter de façon appropriée des tenseurs d’ordre élevé est due au fait que ces données contiennent des informations structurelles multi-voies qui ne peuvent pas être capturées directement par les modèles conventionnels de régression vectorielle ou matricielle. En outre, la très grande dimensionnalité de l’entrée tensorielle produit une énorme quantité de paramètres, ce qui rompt les garanties théoriques des approches de régression classique. De plus, les modèles classiques de régression se sont avérés limités en termes de difficulté d’interprétation, de sensibilité au bruit et d’absence d’unicité. Pour faire face à ces défis, nous étudions une nouvelle classe de modèles de régression, appelés modèles de régression tensor-variable, où les prédicteurs indépendants et (ou) les réponses dépendantes prennent la forme de représentations tensorielles d’ordre élevé. Nous les appliquons également dans de nombreuses applications du monde réel pour vérifier leur efficacité et leur efficacité.With the advancement of modern technologies, high-order tensors are quite widespread and abound in a broad range of applications such as computational neuroscience, computer vision, signal processing and so on. The primary reason that classical regression methods fail to appropriately handle high-order tensors is due to the fact that those data contain multiway structural information which cannot be directly captured by the conventional vector-based or matrix-based regression models, causing substantial information loss during the regression. Furthermore, the ultrahigh dimensionality of tensorial input produces huge amount of parameters, which breaks the theoretical guarantees of classical regression approaches. Additionally, the classical regression models have also been shown to be limited in terms of difficulty of interpretation, sensitivity to noise and absence of uniqueness. To deal with these challenges, we investigate a novel class of regression models, called tensorvariate regression models, where the independent predictors and (or) dependent responses take the form of high-order tensorial representations. We also apply them in numerous real-world applications to verify their efficiency and effectiveness. Concretely, we first introduce hierarchical Tucker tensor regression, a generalized linear tensor regression model that is able to handle potentially much higher order tensor input. Then, we work on online local Gaussian process for tensor-variate regression, an efficient nonlinear GPbased approach that can process large data sets at constant time in a sequential way. Next, we present a computationally efficient online tensor regression algorithm with general tensorial input and output, called incremental higher-order partial least squares, for the setting of infinite time-dependent tensor streams. Thereafter, we propose a super-fast sequential tensor regression framework for general tensor sequences, namely recursive higher-order partial least squares, which addresses issues of limited storage space and fast processing time allowed by dynamic environments. Finally, we introduce kernel-based multiblock tensor partial least squares, a new generalized nonlinear framework that is capable of predicting a set of tensor blocks by merging a set of tensor blocks from different sources with a boosted predictive power

Any-way and Sparse Analyses for Multimodal Fusion and Imaging Genomics

This dissertation aims to develop new algorithms that leverage sparsity and mutual information across data modalities built upon the independent component analysis (ICA) framework to improve the performance of current ICA-based multimodal fusion approaches. These algorithms are further applied to both simulated data and real neuroimaging and genomic data to examine their performance. The identified neuroimaging and genomic patterns can help better delineate the pathology of mental disorders or brain development. To alleviate the signal-background separation difficulties in infomax-decomposed sources for genomic data, we propose a sparse infomax by enhancing a robust sparsity measure, the Hoyer index. Hoyer index is scale-invariant and well suited for ICA frameworks since the scale of decomposed sources is arbitrary. Simulation results demonstrate that sparse infomax increases the component detection accuracy for situations where the source signal-to-background (SBR) ratio is low, particularly for single nucleotide polymorphism (SNP) data. The proposed sparse infomax is further extended into two data modalities as a sparse parallel ICA for applications to imaging genomics in order to investigate the associations between brain imaging and genomics. Simulation results show that sparse parallel ICA outperforms parallel ICA with improved accuracy for structural magnetic resonance imaging (sMRI)-SNP association detection and component spatial map recovery, as well as with enhanced sparsity for sMRI and SNP components under noisy cases. Applying the proposed sparse parallel ICA to fuse the whole-brain sMRI and whole-genome SNP data of 24985 participants in the UK biobank, we identify three stable and replicable sMRI-SNP pairs. The identified sMRI components highlight frontal, parietal, and temporal regions and associate with multiple cognitive measures (with different association strengths in different age groups for the temporal component). Top SNPs in the identified SNP factor are enriched in inflammatory disease and inflammatory response pathways, which also regulate gene expression, isoform percentage, transcription expression, or methylation level in the frontal region, and the regulation effects are significantly enriched. Applying the proposed sparse parallel ICA to imaging genomics in attention-deficit/hyperactivity disorder (ADHD), we identify and replicate one SNP component related to gray matter volume (GMV) alterations in superior and middle frontal gyri underlying working memory deficit in adults and adolescents with ADHD. The association is more significant in ADHD families than controls and stronger in adults and older adolescents than younger ones. The identified SNP component highlights SNPs in long non-coding RNAs (lncRNAs) in chromosome 5 and in several protein-coding genes that are involved in ADHD, such as MEF2C, CADM2, and CADPS2. Top SNPs are enriched in human brain neuron cells and regulate gene expression, isoform percentage, transcription expression, or methylation level in the frontal region. Moreover, to increase the flexibility and robustness in mining multimodal data, we propose aNy-way ICA, which optimizes the entire correlation structure of linked components across any number of modalities via the Gaussian independent vector analysis and simultaneously optimizes independence via separate (parallel) ICAs. Simulation results demonstrate that aNy-way ICA recover sources and loadings, as well as the true covariance patterns with improved accuracy compared to existing multimodal fusion approaches, especially under noisy conditions. Applying the proposed aNy-way ICA to integrate structural MRI, fractal n-back, and emotion identification task functional MRIs collected in the Philadelphia Neurodevelopmental Cohort (PNC), we identify and replicate one linked GMV-threat-2-back component, and the threat and 2-back components are related to intelligence quotient (IQ) score in both discovery and replication samples. Lastly, we extend the proposed aNy-way ICA with a reference constraint to enable prior-guided multimodal fusion. Simulation results show that aNy-way ICA with reference recovers the designed linkages between reference and modalities, cross-modality correlations, as well as loading and component matrices with improved accuracy compared to multi-site canonical correlation analysis with reference (MCCAR)+joint ICA under noisy conditions. Applying aNy-way ICA with reference to supervise structural MRI, fractal n-back, and emotion identification task functional MRIs fusion in PNC with IQ as the reference, we identify and replicate one IQ-related GMV-threat-2-back component, and this component is significantly correlated across modalities in both discovery and replication samples.Ph.D

A Novel Synergistic Model Fusing Electroencephalography and Functional Magnetic Resonance Imaging for Modeling Brain Activities

Study of the human brain is an important and very active area of research. Unraveling the way the human brain works would allow us to better understand, predict and prevent brain related diseases that affect a significant part of the population. Studying the brain response to certain input stimuli can help us determine the involved brain areas and understand the mechanisms that characterize behavioral and psychological traits. In this research work two methods used for the monitoring of brain activities, Electroencephalography (EEG) and functional Magnetic Resonance (fMRI) have been studied for their fusion, in an attempt to bridge together the advantages of each one. In particular, this work has focused in the analysis of a specific type of EEG and fMRI recordings that are related to certain events and capture the brain response under specific experimental conditions. Using spatial features of the EEG we can describe the temporal evolution of the electrical field recorded in the scalp of the head. This work introduces the use of Hidden Markov Models (HMM) for modeling the EEG dynamics. This novel approach is applied for the discrimination of normal and progressive Mild Cognitive Impairment patients with significant results. EEG alone is not able to provide the spatial localization needed to uncover and understand the neural mechanisms and processes of the human brain. Functional Magnetic Resonance imaging (fMRI) provides the means of localizing functional activity, without though, providing the timing details of these activations. Although, at first glance it is apparent that the strengths of these two modalities, EEG and fMRI, complement each other, the fusion of information provided from each one is a challenging task. A novel methodology for fusing EEG spatiotemporal features and fMRI features, based on Canonical Partial Least Squares (CPLS) is presented in this work. A HMM modeling approach is used in order to derive a novel feature-based representation of the EEG signal that characterizes the topographic information of the EEG. We use the HMM model in order to project the EEG data in the Fisher score space and use the Fisher score to describe the dynamics of the EEG topography sequence. The correspondence between this new feature and the fMRI is studied using CPLS. This methodology is applied for extracting features for the classification of a visual task. The results indicate that the proposed methodology is able to capture task related activations that can be used for the classification of mental tasks. Extensions on the proposed models are examined along with future research directions and applications