A Novel Hybrid Dimensionality Reduction Method using Support Vector Machines and Independent Component Analysis

Due to the increasing demand for high dimensional data analysis from various applications such as electrocardiogram signal analysis and gene expression analysis for cancer detection, dimensionality reduction becomes a viable process to extracts essential information from data such that the high-dimensional data can be represented in a more condensed form with much lower dimensionality to both improve classification accuracy and reduce computational complexity. Conventional dimensionality reduction methods can be categorized into stand-alone and hybrid approaches. The stand-alone method utilizes a single criterion from either supervised or unsupervised perspective. On the other hand, the hybrid method integrates both criteria. Compared with a variety of stand-alone dimensionality reduction methods, the hybrid approach is promising as it takes advantage of both the supervised criterion for better classification accuracy and the unsupervised criterion for better data representation, simultaneously. However, several issues always exist that challenge the efficiency of the hybrid approach, including (1) the difficulty in finding a subspace that seamlessly integrates both criteria in a single hybrid framework, (2) the robustness of the performance regarding noisy data, and (3) nonlinear data representation capability. This dissertation presents a new hybrid dimensionality reduction method to seek projection through optimization of both structural risk (supervised criterion) from Support Vector Machine (SVM) and data independence (unsupervised criterion) from Independent Component Analysis (ICA). The projection from SVM directly contributes to classification performance improvement in a supervised perspective whereas maximum independence among features by ICA construct projection indirectly achieving classification accuracy improvement due to better intrinsic data representation in an unsupervised perspective. For linear dimensionality reduction model, I introduce orthogonality to interrelate both projections from SVM and ICA while redundancy removal process eliminates a part of the projection vectors from SVM, leading to more effective dimensionality reduction. The orthogonality-based linear hybrid dimensionality reduction method is extended to uncorrelatedness-based algorithm with nonlinear data representation capability. In the proposed approach, SVM and ICA are integrated into a single framework by the uncorrelated subspace based on kernel implementation. Experimental results show that the proposed approaches give higher classification performance with better robustness in relatively lower dimensions than conventional methods for high-dimensional datasets

An adaptive ensemble learner function via bagging and rank aggregation with applications to high dimensional data.

An ensemble consists of a set of individual predictors whose predictions are combined. Generally, different classification and regression models tend to work well for different types of data and also, it is usually not know which algorithm will be optimal in any given application. In this thesis an ensemble regression function is presented which is adapted from Datta et al. 2010. The ensemble function is constructed by combining bagging and rank aggregation that is capable of changing its performance depending on the type of data that is being used. In the classification approach, the results can be optimized with respect to performance measures such as accuracy, sensitivity, specificity and area under the curve (AUC) whereas in the regression approach, it can be optimized with respect to measures such as mean square error and mean absolute error. The ensemble classifier and ensemble regressor performs at the level of the best individual classifier or regression model. For complex high-dimensional datasets, it may be advisable to combine a number of classification algorithms or regression algorithms rather than using one specific algorithm

A convergence and diversity guided leader selection strategy for many-objective particle swarm optimization

Recently, particle swarm optimizer (PSO) is extended to solve many-objective optimization problems (MaOPs) and becomes a hot research topic in the field of evolutionary computation. Particularly, the leader particle selection (LPS) and the search direction used in a velocity update strategy are two crucial factors in PSOs. However, the LPS strategies for most existing PSOs are not so efficient in high-dimensional objective space, mainly due to the lack of convergence pressure or loss of diversity. In order to address these two issues and improve the performance of PSO in high-dimensional objective space, this paper proposes a convergence and diversity guided leader selection strategy for PSO, denoted as CDLS, in which different leader particles are adaptively selected for each particle based on its corresponding situation of convergence and diversity. In this way, a good tradeoff between the convergence and diversity can be achieved by CDLS. To verify the effectiveness of CDLS, it is embedded into the PSO search process of three well-known PSOs. Furthermore, a new variant of PSO combining with the CDLS strategy, namely PSO/CDLS, is also presented. The experimental results validate the superiority of our proposed CDLS strategy and the effectiveness of PSO/CDLS, when solving numerous MaOPs with regular and irregular Pareto fronts (PFs)