A Novel Hybrid Dimensionality Reduction Method using Support Vector Machines and Independent Component Analysis

Due to the increasing demand for high dimensional data analysis from various applications such as electrocardiogram signal analysis and gene expression analysis for cancer detection, dimensionality reduction becomes a viable process to extracts essential information from data such that the high-dimensional data can be represented in a more condensed form with much lower dimensionality to both improve classification accuracy and reduce computational complexity. Conventional dimensionality reduction methods can be categorized into stand-alone and hybrid approaches. The stand-alone method utilizes a single criterion from either supervised or unsupervised perspective. On the other hand, the hybrid method integrates both criteria. Compared with a variety of stand-alone dimensionality reduction methods, the hybrid approach is promising as it takes advantage of both the supervised criterion for better classification accuracy and the unsupervised criterion for better data representation, simultaneously. However, several issues always exist that challenge the efficiency of the hybrid approach, including (1) the difficulty in finding a subspace that seamlessly integrates both criteria in a single hybrid framework, (2) the robustness of the performance regarding noisy data, and (3) nonlinear data representation capability. This dissertation presents a new hybrid dimensionality reduction method to seek projection through optimization of both structural risk (supervised criterion) from Support Vector Machine (SVM) and data independence (unsupervised criterion) from Independent Component Analysis (ICA). The projection from SVM directly contributes to classification performance improvement in a supervised perspective whereas maximum independence among features by ICA construct projection indirectly achieving classification accuracy improvement due to better intrinsic data representation in an unsupervised perspective. For linear dimensionality reduction model, I introduce orthogonality to interrelate both projections from SVM and ICA while redundancy removal process eliminates a part of the projection vectors from SVM, leading to more effective dimensionality reduction. The orthogonality-based linear hybrid dimensionality reduction method is extended to uncorrelatedness-based algorithm with nonlinear data representation capability. In the proposed approach, SVM and ICA are integrated into a single framework by the uncorrelated subspace based on kernel implementation. Experimental results show that the proposed approaches give higher classification performance with better robustness in relatively lower dimensions than conventional methods for high-dimensional datasets

Visual Scene Understanding by Deep Fisher Discriminant Learning

Modern deep learning has recently revolutionized several fields of classic machine learning and computer vision, such as, scene understanding, natural language processing and machine translation. The substitution of feature hand-crafting with automatic feature learning, provides an excellent opportunity for gaining an in-depth understanding of large-scale data statistics. Deep neural networks generally train models with huge numbers of parameters, facilitating efficient search for optimal and sub-optimal spaces of highly non-convex objective functions. On the other hand, Fisher discriminant analysis has been widely employed to impose class discrepancy, for the sake of segmentation, classification, and recognition tasks. This thesis bridges between contemporary deep learning and classic discriminant analysis, to accommodate some important challenges in visual scene understanding, i.e. semantic segmentation, texture classification, and object recognition. The aim is to accomplish specific tasks in some new high-dimensional spaces, covered by the statistical information of the datasets under study. Inspired by a new formulation of Fisher discriminant analysis, this thesis introduces some novel arrangements of well-known deep learning architectures, to achieve better performances on the targeted missions. The theoretical justifications are based upon a large body of experimental work, and consolidate the contribution of the proposed idea; Deep Fisher Discriminant Learning, to several challenges in visual scene understanding

Dimension-reduction and discrimination of neuronal multi-channel signals

Dimensionsreduktion und Trennung neuronaler Multikanal-Signale

A detection-based pattern recognition framework and its applications

The objective of this dissertation is to present a detection-based pattern recognition framework and demonstrate its applications in automatic speech recognition and broadcast news video story segmentation. Inspired by the studies of modern cognitive psychology and real-world pattern recognition systems, a detection-based pattern recognition framework is proposed to provide an alternative solution for some complicated pattern recognition problems. The primitive features are first detected and the task-specific knowledge hierarchy is constructed level by level; then a variety of heterogeneous information sources are combined together and the high-level context is incorporated as additional information at certain stages. A detection-based framework is a â divide-and-conquerâ design paradigm for pattern recognition problems, which will decompose a conceptually difficult problem into many elementary sub-problems that can be handled directly and reliably. Some information fusion strategies will be employed to integrate the evidence from a lower level to form the evidence at a higher level. Such a fusion procedure continues until reaching the top level. Generally, a detection-based framework has many advantages: (1) more flexibility in both detector design and fusion strategies, as these two parts can be optimized separately; (2) parallel and distributed computational components in primitive feature detection. In such a component-based framework, any primitive component can be replaced by a new one while other components remain unchanged; (3) incremental information integration; (4) high level context information as additional information sources, which can be combined with bottom-up processing at any stage. This dissertation presents the basic principles, criteria, and techniques for detector design and hypothesis verification based on the statistical detection and decision theory. In addition, evidence fusion strategies were investigated in this dissertation. Several novel detection algorithms and evidence fusion methods were proposed and their effectiveness was justified in automatic speech recognition and broadcast news video segmentation system. We believe such a detection-based framework can be employed in more applications in the future.Ph.D.Committee Chair: Lee, Chin-Hui; Committee Member: Clements, Mark; Committee Member: Ghovanloo, Maysam; Committee Member: Romberg, Justin; Committee Member: Yuan, Min

Regularized Risk Prediction Models in Subject/Patient Analytics in a Time to Event Setting

This thesis comprises of five investigations and focuses on the use of risk prediction modelling from a computational statistics and machine learning perspective, with applications in subject (e.g. gym user, patient) analytics in a time to event setting. The work was conducted in collaboration with eGym and UCL Hospitals (UCLH). A variety of computational statistics (e.g. logistic lasso) and machine learning based risk prediction methods are applied ranging from kernel methods, ensemble methods and decision trees from both a classification and survival perspective. The thesis is concerned with modelling gym user behaviour and predicting treatment times and types. The underlying goal of this thesis is to develop generalizable and useful models to predict gym user behaviour and patient treatment times. This is what leads us to our methodological work in chapter 6. This thesis conducts the following investigations. 1. Weibull full likelihood implementation The first investigation involves conducting an implementation of a Weibull full likelihood survival model in R. The aim of this investigation is to build the Weibull distribution proportional hazards model, which is formulated via the log likelihood. Then we apply this model to simulated data to see whether the model can reveal the real pattern of the data. The results prove that from the synthetic data the model we build in R can unearth the parameters and the coefficients from which we generate the data. 2. Predicting gym user behaviour through churn and visits The second investigation consisting of two sub-investigations considers the use of time to event models to predict gym user behaviour and churn. The data set has been provided by the Gym Equipment manufacturer eGym. The first sub-investigation considers if it is possible, we can predict whether or not a user will churn, using a range of methods across computational statistics and machine learning, from logistic regression to survival random forests. Our findings indicate that with demographics alone we are unable to produce machine learning models that outperform a baseline learner. This tells us that we are unable to predict right at the beginning, whether or not a user will churn. However, when we apply machine learning based survival models including elastic net Cox and Cox Boosting, we are able to outperform the baseline. This sub-investigation serves as an introduction to considering gym user churn in a time to event setting through both classification and survival models. In the second sub-investigation, we then apply risk prediction modelling in predicting gym user visits via a moving window model, we find we are marginally able to outperform the majority vote baseline in some settings. 3 3. Predicting patient treatment times and treatment types for patient rehabilitative care The third investigation, also consisting of two sub-investigations, concerns the use of time to event modelling to predict patient treatment times and treatment types for patient rehabilitative care. The underlying goal is to help design treatment plans aimed at helping patients return to work by predicting the required combination of treatment time and treatment types required for each patient. The data has been provided by UCLH. All patients in the data set have been eventually discharged from the treatment programme. The aim of the first sub-investigation is to predict how much treatment time the patients required before they were discharged and which patients are more likely to take longer. We model this problem using regression and survival analysis, methods used range from generalized additive models to Cox boosting. Our results show that, using demographic variables we are able to outperform the baseline. In the second sub-investigation, we utilise risk prediction models, such as logistic regression and Adaboosting to predict treatment types based on demographics. We are able to outperform the baseline for some treatments in a deterministic setting but not in a probabilistic setting. 4. Regularization problems in gym user/patient setting As alluded, in both our application settings our model performances are mixed. Our aim therefore is to investigate how we can potentially improve our model performance and usefulness. This is what motivates our methodological studies: improving our model performance via hyper-parameter tuning based on the relevant loss function. We begin our investigation by using F1, Brier score and net benefit as the scoring functions for parameter tuning to build LASSO models. We then run the models on the gym user data and hospital data and compare the performance outputs from modelling. We find we are able to outperform the conventional LASSO models in terms of F1, Brier score and net benefit when using them as tuning functions, respectively. The different LASSO models provide different variable selections and insights. Then we use the integrated Brier score to turn the parameters of Cox proportional hazards LASSO models in a survival setting. Compared with the conventional performance measure - Concordance index, the integrated Brier score reflects better the error measure overall time. We find that by tuning parameters for the integrated Brier score we are able to obtain better integrated Brier score performance and different variable selections. We also examine whether the integrated Brier score is not only useful for improving survival performance at all times but at specific times too. We apply the Cox proportional hazards LASSO models with integrated Brier score and Concordance index as the scoring functions to the gym user and hospital data sets. The results show the models can better perform on the corresponding loss functions but the integrated Brier score LASSO model doesn’t guarantee better performance at a specific time. Finally, we extend our methodology to more modern machine learning methods such as support vector machines. We use F1 score, Brier score and net benefit as scoring functions to turn the parameters and C 4 of SVMs and run the models on the gym user data and hospital data. The results show they only slightly outperform the conventional model and are specifically poor in the deterministic setting due to the data imbalance. Contributions to Science This thesis makes the following contributions to science. 1. Applies logistic regression, linear discriminant analysis, support vector machines and random forests to predict the gym user attendance and churn. 2. Introduces the idea of comparing gym user prediction models to a majority vote baseline. 3. Introduces moving window prediction models for gym user visit prediction. 4. Discovers the relationship between patient demographics and rehabilitative care treatment times. 5. Introduces machine learning and computational statistics to predict patient treatment times and types for neurological rehabilitation patients. 6. Introduces the use of the F1, Brier score and in particular the net benefit LASSO models to a gym user churn prediction and a treatment type prediction. 7. Introduces the use of the integrated Brier score for tuning Cox LASSO models. 8. Extends the idea of parameter turning via the F1, Brier score and net benefit to modern machine learning methods