Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning

A central problem in unsupervised deep learning is how to find useful representations of high-dimensional data, sometimes called "disentanglement". Most approaches are heuristic and lack a proper theoretical foundation. In linear representation learning, independent component analysis (ICA) has been successful in many applications areas, and it is principled, i.e. based on a well-defined probabilistic model. However, extension of ICA to the nonlinear case has been problematic due to the lack of identifiability, i.e. uniqueness of the representation. Recently, nonlinear extensions that utilize temporal structure or some auxiliary information have been proposed. Such models are in fact identifiable, and consequently, an increasing number of algorithms have been developed. In particular, some self-supervised algorithms can be shown to estimate nonlinear ICA, even though they have initially been proposed from heuristic perspectives. This paper reviews the state-of-the-art of nonlinear ICA theory and algorithms

Role of independent component analysis in intelligent ECG signal processing

The Electrocardiogram (ECG) reflects the activities and the attributes of the human heart and reveals very important hidden information in its structure. The information is extracted by means of ECG signal analysis to gain insights that are very crucial in explaining and identifying various pathological conditions. The feature extraction process can be accomplished directly by an expert through, visual inspection of ECGs printed on paper or displayed on a screen. However, the complexity and the time taken for the ECG signals to be visually inspected and manually analysed means that it‟s a very tedious task thus yielding limited descriptions. In addition, a manual ECG analysis is always prone to errors: human oversights. Moreover ECG signal processing has become a prevalent and effective tool for research and clinical practices. A typical computer based ECG analysis system includes a signal preprocessing, beats detection and feature extraction stages, followed by classification.Automatic identification of arrhythmias from the ECG is one important biomedical application of pattern recognition. This thesis focuses on ECG signal processing using Independent Component Analysis (ICA), which has received increasing attention as a signal conditioning and feature extraction technique for biomedical application. Long term ECG monitoring is often required to reliably identify the arrhythmia. Motion induced artefacts are particularly common in ambulatory and Holter recordings, which are difficult to remove with conventional filters due to their similarity to the shape of ectopic xiiibeats. Feature selection has always been an important step towards more accurate, reliable and speedy pattern recognition. Better feature spaces are also sought after in ECG pattern recognition applications. Two new algorithms are proposed, developed and validated in this thesis, one for removing non-trivial noises in ECGs using the ICA and the other deploys the ICA extracted features to improve recognition of arrhythmias. Firstly, independent component analysis has been studiedand found effective in this PhD project to separate out motion induced artefacts in ECGs, the independent component corresponding to noise is then removed from the ECG according to kurtosis and correlation measurement.The second algorithm has been developed for ECG feature extraction, in which the independent component analysis has been used to obtain a set of features, or basis functions of the ECG signals generated hypothetically by different parts of the heart during the normal and arrhythmic cardiac cycle. ECGs are then classified based on the basis functions along with other time domain features. The selection of the appropriate feature set for classifier has been found important for better performance and quicker response. Artificial neural networks based pattern recognition engines are used to perform final classification to measure the performance of ICA extracted features and effectiveness of the ICA based artefacts reduction algorithm.The motion artefacts are effectively removed from the ECG signal which is shown by beat detection on noisy and cleaned ECG signals after ICA processing. Using the ICA extracted feature sets classification of ECG arrhythmia into eight classes with fewer independent components and very high classification accuracy is achieved

Unsupervised methods in multilingual and multimodal semantic modeling

In the first part of this project, independent component analysis has been applied to extract word clusters from two Farsi corpora. Both word-document and word-context matrices have been considered to extract such clusters. The application of ICA on the word-document matrices extracted from these two corpora led to the detection of syntagmatic word clusters, while the utilization of word-context matrix resulted in the extraction of both syntagmatic and paradigmatic word clusters. Furthermore, we have discussed some potential benefits of this automatically extracted thesaurus. In such a thesaurus, a word is defined by some other words without being connected to the outer physical objects. In order to fill such a gap, symbol grounding has been proposed by philosophers as a mechanism which might connect words to their physical referents. From their point of view, if words are properly connected to their referents, their meaning might be realized. Once this objective is achieved, a new promising horizon would open in the realm of artificial intelligence. In the second part of the project, we have offered a simple but novel method for grounding words based on the features coming from the visual modality. Firstly, indexical grounding is implemented. In this naïve symbol grounding method, a word is characterized using video indexes as its context. Secondly, such indexical word vectors have been normalized according to the features calculated for motion videos. This multimodal fusion has been referred to as the pattern grounding. In addition, the indexical word vectors have been normalized using some randomly generated data instead of the original motion features. This third case was called randomized grounding. These three cases of symbol grounding have been compared in terms of the performance of translation. Besides that, word clusters have been excerpted by comparing the vector distances and from the dendrograms generated using an agglomerative hierarchical clustering method. We have observed that pattern grounding exceled the indexical grounding in the translation of the motion annotated words, while randomized grounding has deteriorated the translation significantly. Moreover, pattern grounding culminated in the formation of clusters in which a word fit semantically to the other members, while using the indexical grounding, some of the closely related words dispersed into arbitrary clusters

Iterative issues of ICA, quality of separation and number of sources : a study for biosignal applications

This thesis has evaluated the use of Independent Component Analysis (ICA) on Surface Electromyography (sEMG), focusing on the biosignal applications. This research has identified and addressed the following four issues related to the use of ICA for biosignals: • The iterative nature of ICA • The order and magnitude ambiguity problems of ICA • Estimation of number of sources based on dependency and independency nature of the signals • Source separation for non-quadratic ICA (undercomplete and overcomplete) This research first establishes the applicability of ICA for sEMG and also identifies the shortcomings related to order and magnitude ambiguity. It has then developed, a mitigation strategy for these issues by using a single unmixing matrix and neural network weight matrix corresponding to the specific user. The research reports experimental verification of the technique and also the investigation of the impact of inter-subject and inter-experimental variations. The results demonstrate that while using sEMG without separation gives only 60% accuracy, and sEMG separated using traditional ICA gives an accuracy of 65%, this approach gives an accuracy of 99% for the same experimental data. Besides the marked improvement in accuracy, the other advantages of such a system are that it is suitable for real time operations and is easy to train by a lay user. The second part of this thesis reports research conducted to evaluate the use of ICA for the separation of bioelectric signals when the number of active sources may not be known. The work proposes the use of value of the determinant of the Global matrix generated using sparse sub band ICA for identifying the number of active sources. The results indicate that the technique is successful in identifying the number of active muscles for complex hand gestures. The results support the applications such as human computer interface. This thesis has also developed a method of determining the number of independent sources in a given mixture and has also demonstrated that using this information, it is possible to separate the signals in an undercomplete situation and reduce the redundancy in the data using standard ICA methods. The experimental verification has demonstrated that the quality of separation using this method is better than other techniques such as Principal Component Analysis (PCA) and selective PCA. This has number of applications such as audio separation and sensor networks