Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning

A central problem in unsupervised deep learning is how to find useful representations of high-dimensional data, sometimes called "disentanglement". Most approaches are heuristic and lack a proper theoretical foundation. In linear representation learning, independent component analysis (ICA) has been successful in many applications areas, and it is principled, i.e. based on a well-defined probabilistic model. However, extension of ICA to the nonlinear case has been problematic due to the lack of identifiability, i.e. uniqueness of the representation. Recently, nonlinear extensions that utilize temporal structure or some auxiliary information have been proposed. Such models are in fact identifiable, and consequently, an increasing number of algorithms have been developed. In particular, some self-supervised algorithms can be shown to estimate nonlinear ICA, even though they have initially been proposed from heuristic perspectives. This paper reviews the state-of-the-art of nonlinear ICA theory and algorithms

Nonparametric Independent Component Analysis for the Sources with Mixed Spectra

Independent component analysis (ICA) is a blind source separation method to recover source signals of interest from their mixtures. Most existing ICA procedures assume independent sampling. Second-order-statistics-based source separation methods have been developed based on parametric time series models for the mixtures from the autocorrelated sources. However, the second-order-statistics-based methods cannot separate the sources accurately when the sources have temporal autocorrelations with mixed spectra. To address this issue, we propose a new ICA method by estimating spectral density functions and line spectra of the source signals using cubic splines and indicator functions, respectively. The mixed spectra and the mixing matrix are estimated by maximizing the Whittle likelihood function. We illustrate the performance of the proposed method through simulation experiments and an EEG data application. The numerical results indicate that our approach outperforms existing ICA methods, including SOBI algorithms. In addition, we investigate the asymptotic behavior of the proposed method.Comment: 27 pages, 10 figure

Independent component analysis on spectral domain

Independent component analysis (ICA) is an effective data-driven method for blind source separation. It has been successfully applied to separate source signals of interest from their mixtures. Most existing ICA procedures are carried out by relying solely on the estimation of the marginal density functions, either parametrically or nonparametrically. In many applications, correlation structures within each source also play an important role besides the marginal distributions. One important example is functional magnetic resonance imaging (fMRI) analysis where the brain-function-related signals are temporally correlated. In this thesis, we propose two novel ICA algorithms that fully exploit the correlation structures within the source signals through spectral density estimation. Our methodology development is two-fold: 1) ICA for auto-correlated sources via parametric spectral density estimation (cICA-YW); 2) ICA for sources with mixed spectra via nonparametric spectral density estimation and atom detection (cICA-LSP). The cICA-YW focuses on the sources with autocorrelation and is implemented using spectral density functions from frequently used time series models such as autoregressive moving average (ARMA) processes. The time series parameters and the mixing matrix are estimated via maximizing the Whittle likelihood function. We illustrate the performance of the proposed method through extensive simulation studies and a real fMRI application. The numerical results indicate that our approach outperforms several popular methods including the most widely used fastICA algorithm. We also establish the sampling properties of the proposed method. For the cICA-LSP, we consider the case of sources with possibly mixed spectra, where ARMA estimates are often unstable. Specifically, we propose to estimate the spectral density functions and the line spectra of the source signals using cubic splines and indicator functions, respectively. The mixed spectra and the mixing matrix are estimated via maximizing the Whittle likelihood function. We illustrate the performance of the proposed method through extensive simulation studies.Doctor of Philosoph

Efficient Multiband Algorithms for Blind Source Separation

The problem of blind separation refers to recovering original signals, called source signals, from the mixed signals, called observation signals, in a reverberant environment. The mixture is a function of a sequence of original speech signals mixed in a reverberant room. The objective is to separate mixed signals to obtain the original signals without degradation and without prior information of the features of the sources. The strategy used to achieve this objective is to use multiple bands that work at a lower rate, have less computational cost and a quicker convergence than the conventional scheme. Our motivation is the competitive results of unequal-passbands scheme applications, in terms of the convergence speed. The objective of this research is to improve unequal-passbands schemes by improving the speed of convergence and reducing the computational cost. The first proposed work is a novel maximally decimated unequal-passbands scheme.This scheme uses multiple bands that make it work at a reduced sampling rate, and low computational cost. An adaptation approach is derived with an adaptation step that improved the convergence speed. The performance of the proposed scheme was measured in different ways. First, the mean square errors of various bands are measured and the results are compared to a maximally decimated equal-passbands scheme, which is currently the best performing method. The results show that the proposed scheme has a faster convergence rate than the maximally decimated equal-passbands scheme. Second, when the scheme is tested for white and coloured inputs using a low number of bands, it does not yield good results; but when the number of bands is increased, the speed of convergence is enhanced. Third, the scheme is tested for quick changes. It is shown that the performance of the proposed scheme is similar to that of the equal-passbands scheme. Fourth, the scheme is also tested in a stationary state. The experimental results confirm the theoretical work. For more challenging scenarios, an unequal-passbands scheme with over-sampled decimation is proposed; the greater number of bands, the more efficient the separation. The results are compared to the currently best performing method. Second, an experimental comparison is made between the proposed multiband scheme and the conventional scheme. The results show that the convergence speed and the signal-to-interference ratio of the proposed scheme are higher than that of the conventional scheme, and the computation cost is lower than that of the conventional scheme

Etude de la propagation de l‟activité électrique utérine dans une optique clinique : Application a la détection des menaces d‟accouchement prématuré

Uterine contractions are essentially controlled by two physiological phenomena: cell excitability and propagation of uterine electrical activity probably related to high and low frequencies of uterine electromyogram, called electrohysterogram -EHG-, respectively. All previous studies have been focused on extracting parameters from the high frequency part and did not show a satisfied potential for clinical application. The objective of this thesis is the analysis of the propagation EHG signals of during pregnancy and labor in the view of extracting tool for clinical application. A novelty of our thesis is the multichannel recordings by using 4x4 electrodes matrix posed on the woman abdomen. Monovariate analysis was aimed to investigate the nonlinear characteristics of EHG signals. Bivariate and multivariate analyses have been done to analyze the propagation of the EHG signals by detecting the connectivity between the signals. An increase of the nonlinearity associated by amplitude synchronization and phase desynchronization were detected. Results indicate a highest EHG propagation during labor than pregnancy and an increase of this propagation with the week of gestations. The results show the high potential of propagation‟s parameters in clinical point of view such as labor detection and then preterm labor prediction. We proposed novel combination of Blind Source Separation and empirical mode decomposition to denoise monopolar EHG as a possible way to increase the classification rate of pregnancy and labor.Les contractions utérines sont contrôlées par deux phénomènes physiologiques: l'excitabilité cellulaire et la propagation de l'activité électrique utérine probablement liées aux hautes et basses fréquences de l‟electrohysterograme (EHG) respectivement. Toutes les études précédentes ont porté sur l'extraction de paramètres de la partie haute fréquence et n'ont pas montré un potentiel satisfait pour l'application clinique. L'objectif de cette thèse est l'analyse de propagation de l'EHG pendant la grossesse et le travail dans la vue de l'extraction des outils pour une application clinique. Une des nouveautés de la thèse est l‟enregistrement multicanaux à l'aide d‟une matrice d'électrodes 4x4 posée sur l'abdomen de la femme. Analyse monovariés visait à étudier les caractéristiques non linéaires des signaux EHG, analyses bivariées et multivariées ont été effectuées pour analyser la propagation des signaux EHG par la détection de la connectivité entre les signaux. Une augmentation de la non- linéarité associée par une synchronisation en amplitude et de désynchronisation en phase a été détectée. Les résultats indiquent plus de propagation au cours du travail que la grossesse et une augmentation de cette propagation avec les semaines de gestations. Les résultats montrent le potentiel élevé de paramètres de propagation dans le point de vue clinique tel que la détection du travail et de prédiction du travail prématuré. Finalement, nous avons proposé une nouvelle combinaison entre Séparation Aveugles de Sources et la Décomposition en Modes Empiriques pour débruiter les signaux EHG monopolaires comme un moyen possible d'augmenter le taux de classification de signaux grossesse et l'accouchement

Analysis of the propagation of uterine electrical activity applied to predict preterm labor

There are many open questions concerning the functioning of the human uterus. One of these open questions concerns exactly how the uterus operates as an organ to perform the very organized act of contracting in a synchronized fashion to expulse a new human into this world. If we don‟t understand how it works when it is working normally, it is obvious that we will not be as capable of intervening or preventing when, sometimes with tragic consequences, it does not do its job properly and a child is born before it is ready. The aim of our research is to be able to understand what the electrical activity of the uterus can tell us about the risk of premature birth, to understand better how the uterus works and to benefit from these understanding to find tool that can be used for labor detection and prediction of preterm labor. This idea of using the externally detected electrical activity of the uterus (electrohysterogram or EHG) to predict preterm labor is not new and lot of work has already been put into it. The novel approach in this work is not to use the signal collected from one or two isolated places on the expectant mother‟s abdomen but to map the propagation of the signals and to investigate the auto organization of the contractions. We therefore use a matrix of electrodes to give us a much more complete picture of the organization and operation of the uterus as pregnancy reaches its conclusion. Labor is the physiologic process by which a fetus is expelled from the uterus to the outside world and is defined as regular uterine contractions accompanied by cervical effacement and dilatation. In the normal labor, the uterine contractions and cervix dilatation are preceded by biochemical changes in the cervical connective tissue.Il reste beaucoup de questions ouvertes concernant le fonctionnement de l'utérus humain. L'une de ces questions est comment l'utérus fonctionne en tant qu‟organe organisé pour générer une contraction synchrone et expulser un nouvel être humain dans ce monde ? Si nous ne comprenons pas comment l‟utérus fonctionne, quand il fonctionne normalement, il est évident que nous ne serons pas en mesure d'intervenir ou de prévoir quand, avec parfois des conséquences tragiques, il ne fait pas son travail correctement et qu‟un enfant nait avant d‟être prêt ! Le but de notre recherche est de comprendre ce que l'activité électrique de l'utérus peut nous apporter sur la prévention du risque de naissance prématurée, de mieux comprendre comment fonctionne l'utérus et de bénéficier de ces connaissances pour développer un outil qui peut être utilisé pour la détection de l‟accouchement et la prédiction du travail prématuré. Cette idée d'utiliser l'activité électrique détectée à la surface de l‟abdomen (ou électrohystérogramme EHG) pour prédire un accouchement prématuré n'est pas nouvelle et beaucoup de travaux ont déjà été mis en oeuvre. La nouvelle approche dans ce travail n‟est pas d‟utiliser le signal recueilli par un ou deux endroits isolés sur l'abdomen de la future mère, mais de cartographier la propagation des signaux et d‟explorer l'auto organisation des contractions. Nous utilisons donc une matrice d'électrodes pour nous donner une image beaucoup plus complète de l'organisation et du fonctionnement de l'utérus. L‟accouchement est le processus physiologique par lequel le foetus est expulsé de l'utérus vers le monde extérieur. Il est défini comme la survenue de contractions utérines régulières accompagnées de l'effacement du col et de la dilatation cervicale. Dans le travail normal, les contractions de l'utérus et la dilatation du col sont précédées par des changements biochimiques du tissu conjonctif du col utérin

Etude de la propagation de l’activité électrique utérine dans une optique clinique: Application à la détection des menaces d’accouchement prématuré.

Uterine contractions are essentially controlled by two physiological phenomena: cell excitability and propagation of uterine electrical activity probably related to high and low frequencies of uterine electromyogram, called electrohysterogram -EHG-, respectively. All previous studies have been focused on extracting parameters from the high frequency part and did not show a satisfied potential for clinical application. The objective of this thesis is the analysis of the propagation EHG signals of during pregnancy and labor in the view of extracting tool for clinical application. A novelty of our thesis is the multichannel recordings by using 4x4 electrodes matrix posed on the woman abdomen. Monovariate analysis was aimed to investigate the nonlinear characteristics of EHG signals. Bivariate and multivariate analyses have been done to analyze the propagation of the EHG signals by detecting the connectivity between the signals. An increase of the nonlinearity associated by amplitude synchronization and phase desynchronization were detected. Results indicate a highest EHG propagation during labor than pregnancy and an increase of this propagation with the week of gestations. The results show the high potential of propagation’s parameters in clinical point of view such as labor detection and then preterm labor prediction. We proposed novel combination of Blind Source Separation and empirical mode decomposition to denoise monopolar EHG as a possible way to increase the classification rate of pregnancy and labor.Les contractions utérines sont contrôlées par deux phénomènes physiologiques: l'excitabilité cellulaire et la propagation de l'activité électrique utérine probablement liées aux hautes et basses fréquences de l’electrohysterograme (EHG) respectivement. Toutes les études précédentes ont porté sur l'extraction de paramètres de la partie haute fréquence et n'ont pas montré un potentiel satisfait pour l'application clinique. L'objectif de cette thèse est l'analyse de propagation de l'EHG pendant la grossesse et le travail dans la vue de l'extraction des outils pour une application clinique. Une des nouveautés de la thèse est l’enregistrement multicanaux à l'aide d’une matrice d'électrodes 4x4 posée sur l'abdomen de la femme. Analyse monovariés visait à étudier les caractéristiques non linéaires des signaux EHG, analyses bivariées et multivariées ont été effectuées pour analyser la propagation des signaux EHG par la détection de la connectivité entre les signaux. Une augmentation de la non-linéarité associée par une synchronisation en amplitude et de désynchronisation en phase a été détectée. Les résultats indiquent plus de propagation au cours du travail que la grossesse et une augmentation de cette propagation avec les semaines de gestations. Les résultats montrent le potentiel élevé de paramètres de propagation dans le point de vue clinique tel que la détection du travail et de prédiction du travail prématuré. Finalement, nous avons proposé une nouvelle combinaison entre Séparation Aveugles de Sources et la Décomposition en Modes Empiriques pour débruiter les signaux EHG monopolaires comme un moyen possible d'augmenter le taux de classification de signaux grossesse et l'accouchement

Untangling hotel industry’s inefficiency: An SFA approach applied to a renowned Portuguese hotel chain

The present paper explores the technical efficiency of four hotels from Teixeira Duarte Group - a renowned Portuguese hotel chain. An efficiency ranking is established from these four hotel units located in Portugal using Stochastic Frontier Analysis. This methodology allows to discriminate between measurement error and systematic inefficiencies in the estimation process enabling to investigate the main inefficiency causes. Several suggestions concerning efficiency improvement are undertaken for each hotel studied.info:eu-repo/semantics/publishedVersio

Causal Discovery from Temporal Data: An Overview and New Perspectives

Temporal data, representing chronological observations of complex systems, has always been a typical data structure that can be widely generated by many domains, such as industry, medicine and finance. Analyzing this type of data is extremely valuable for various applications. Thus, different temporal data analysis tasks, eg, classification, clustering and prediction, have been proposed in the past decades. Among them, causal discovery, learning the causal relations from temporal data, is considered an interesting yet critical task and has attracted much research attention. Existing casual discovery works can be divided into two highly correlated categories according to whether the temporal data is calibrated, ie, multivariate time series casual discovery, and event sequence casual discovery. However, most previous surveys are only focused on the time series casual discovery and ignore the second category. In this paper, we specify the correlation between the two categories and provide a systematical overview of existing solutions. Furthermore, we provide public datasets, evaluation metrics and new perspectives for temporal data casual discovery.Comment: 52 pages, 6 figure

Metric Gaussian variational inference

One main result of this dissertation is the development of Metric Gaussian Variational Inference (MGVI), a method to perform approximate inference in extremely high dimensions and for complex probabilistic models. The problem with high-dimensional and complex models is twofold. Fist, to capture the true posterior distribution accurately, a sufficiently rich approximation for it is required. Second, the number of parameters to express this richness scales dramatically with the number of model parameters. For example, explicitly expressing the correlation between all model parameters requires their squared number of correlation coefficients. In settings with millions of model parameter, this is unfeasible. MGVI overcomes this limitation by replacing the explicit covariance with an implicit approximation, which does not have to be stored and is accessed via samples. This procedure scales linearly with the problem size and allows to account for the full correlations in even extremely large problems. This makes it also applicable to significantly more complex setups. MGVI enabled a series of ambitious signal reconstructions by me and others, which will be showcased. This involves a time- and frequency-resolved reconstruction of the shadow around the black hole M87* using data provided by the Event Horizon Telescope Collaboration, a three-dimensional tomographic reconstruction of interstellar dust within 300pc around the sun from Gaia starlight-absorption and parallax data, novel medical imaging methods for computed tomography, an all-sky Faraday rotation map, combining distinct data sources, and simultaneous calibration and imaging with a radio-interferometer. The second main result is an an approach to use several, independently trained and deep neural networks to reason on complex tasks. Deep learning allows to capture abstract concepts by extracting them from large amounts of training data, which alleviates the necessity of an explicit mathematical formulation. Here a generative neural network is used as a prior distribution and certain properties are imposed via classification and regression networks. The inference is then performed in terms of the latent variables of the generator, which is done using MGVI and other methods. This allows to flexibly answer novel questions without having to re-train any neural network and to come up with novel answers through Bayesian reasoning. This novel approach of Bayesian reasoning with neural networks can also be combined with conventional measurement data