L2-norm multiple kernel learning and its application to biomedical data fusion

<p>Abstract</p> <p>Background</p> <p>This paper introduces the notion of optimizing different norms in the dual problem of support vector machines with multiple kernels. The selection of norms yields different extensions of multiple kernel learning (MKL) such as <it>L</it>_∞, <it>L</it>₁, and <it>L</it>₂MKL. In particular, <it>L</it>₂MKL is a novel method that leads to non-sparse optimal kernel coefficients, which is different from the sparse kernel coefficients optimized by the existing <it>L</it>_∞MKL method. In real biomedical applications, <it>L</it>₂MKL may have more advantages over sparse integration method for thoroughly combining complementary information in heterogeneous data sources.</p> <p>Results</p> <p>We provide a theoretical analysis of the relationship between the <it>L</it>₂optimization of kernels in the dual problem with the <it>L</it>₂coefficient regularization in the primal problem. Understanding the dual <it>L</it>₂problem grants a unified view on MKL and enables us to extend the <it>L</it>₂method to a wide range of machine learning problems. We implement <it>L</it>₂MKL for ranking and classification problems and compare its performance with the sparse <it>L</it>_∞and the averaging <it>L</it>₁MKL methods. The experiments are carried out on six real biomedical data sets and two large scale UCI data sets. <it>L</it>₂MKL yields better performance on most of the benchmark data sets. In particular, we propose a novel <it>L</it>₂MKL least squares support vector machine (LSSVM) algorithm, which is shown to be an efficient and promising classifier for large scale data sets processing.</p> <p>Conclusions</p> <p>This paper extends the statistical framework of genomic data fusion based on MKL. Allowing non-sparse weights on the data sources is an attractive option in settings where we believe most data sources to be relevant to the problem at hand and want to avoid a "winner-takes-all" effect seen in <it>L</it>_∞MKL, which can be detrimental to the performance in prospective studies. The notion of optimizing <it>L</it>₂kernels can be straightforwardly extended to ranking, classification, regression, and clustering algorithms. To tackle the computational burden of MKL, this paper proposes several novel LSSVM based MKL algorithms. Systematic comparison on real data sets shows that LSSVM MKL has comparable performance as the conventional SVM MKL algorithms. Moreover, large scale numerical experiments indicate that when cast as semi-infinite programming, LSSVM MKL can be solved more efficiently than SVM MKL.</p> <p>Availability</p> <p>The MATLAB code of algorithms implemented in this paper is downloadable from <url>http://homes.esat.kuleuven.be/~sistawww/bioi/syu/l2lssvm.html</url>.</p

Least squares support vector machine with self-organizing multiple kernel learning and sparsity

© 2018 In recent years, least squares support vector machines (LSSVMs) with various kernel functions have been widely used in the field of machine learning. However, the selection of kernel functions is often ignored in practice. In this paper, an improved LSSVM method based on self-organizing multiple kernel learning is proposed for black-box problems. To strengthen the generalization ability of the LSSVM, some appropriate kernel functions are selected and the corresponding model parameters are optimized using a differential evolution algorithm based on an improved mutation strategy. Due to the large computation cost, a sparse selection strategy is developed to extract useful data and remove redundant data without loss of accuracy. To demonstrate the effectiveness of the proposed method, some benchmark problems from the UCI machine learning repository are tested. The results show that the proposed method performs better than other state-of-the-art methods. In addition, to verify the practicability of the proposed method, it is applied to a real-world converter steelmaking process. The results illustrate that the proposed model can precisely predict the molten steel quality and satisfy the actual production demand

Fault Diagnosis and Failure Prognostics of Lithium-ion Battery based on Least Squares Support Vector Machine and Memory Particle Filter Framework

123456A novel data driven approach is developed for fault diagnosis and remaining useful life (RUL) prognostics for lithium-ion batteries using Least Square Support Vector Machine (LS-SVM) and Memory-Particle Filter (M-PF). Unlike traditional data-driven models for capacity fault diagnosis and failure prognosis, which require multidimensional physical characteristics, the proposed algorithm uses only two variables: Energy Efficiency (EE), and Work Temperature. The aim of this novel framework is to improve the accuracy of incipient and abrupt faults diagnosis and failure prognosis. First, the LSSVM is used to generate residual signal based on capacity fade trends of the Li-ion batteries. Second, adaptive threshold model is developed based on several factors including input, output model error, disturbance, and drift parameter. The adaptive threshold is used to tackle the shortcoming of a fixed threshold. Third, the M-PF is proposed as the new method for failure prognostic to determine Remaining Useful Life (RUL). The M-PF is based on the assumption of the availability of real-time observation and historical data, where the historical failure data can be used instead of the physical failure model within the particle filter. The feasibility of the framework is validated using Li-ion battery prognostic data obtained from the National Aeronautic and Space Administration (NASA) Ames Prognostic Center of Excellence (PCoE). The experimental results show the following: (1) fewer data dimensions for the input data are required compared to traditional empirical models; (2) the proposed diagnostic approach provides an effective way of diagnosing Li-ion battery fault; (3) the proposed prognostic approach can predict the RUL of Li-ion batteries with small error, and has high prediction accuracy; and, (4) the proposed prognostic approach shows that historical failure data can be used instead of a physical failure model in the particle filter

High-Order Inference, Ranking, and Regularization Path for Structured SVM

This thesis develops novel methods to enable the use of structured prediction in computer vision and medical imaging. Specifically, our contributions are four fold. First, we propose a new family of high-order potentials that encourage parsimony in the labeling, and enable its use by designing an accurate graph cuts based algorithm to minimize the corresponding energy function. Second, we show how the average precision SVM formulation can be extended to incorporate high-order information for ranking. Third, we propose a novel regularization path algorithm for structured SVM. Fourth, we show how the weakly supervised framework of latent SVM can be employed to learn the parameters for the challenging deformable registration problem.In more detail, the first part of the thesis investigates the high-order inference problem. Specifically, we present a novel family of discrete energy minimization problems, which we call parsimonious labeling. It is a natural generalization of the well known metric labeling problems for high-order potentials. In addition to this, we propose a generalization of the Pn-Potts model, which we call Hierarchical Pn-Potts model. In the end, we propose parallelizable move making algorithms with very strong multiplicative bounds for the optimization of the hierarchical Pn-Potts model and the parsimonious labeling.Second part of the thesis investigates the ranking problem while using high-order information. Specifically, we introduce two alternate frameworks to incorporate high-order information for the ranking tasks. The first framework, which we call high-order binary SVM (HOB-SVM), optimizes a convex upperbound on weighted 0-1 loss while incorporating high-order information using joint feature map. The rank list for the HOB-SVM is obtained by sorting samples using max-marginals based scores. The second framework, which we call high-order AP-SVM (HOAP-SVM), takes its inspiration from AP-SVM and HOB-SVM (our first framework). Similar to AP-SVM, it optimizes upper bound on average precision. However, unlike AP-SVM and similar to HOB-SVM, it can also encode high-order information. The main disadvantage of HOAP-SVM is that estimating its parameters requires solving a difference-of-convex program. We show how a local optimum of the HOAP-SVM learning problem can be computed efficiently by the concave-convex procedure. Using standard datasets, we empirically demonstrate that HOAP-SVM outperforms the baselines by effectively utilizing high-order information while optimizing the correct loss function.In the third part of the thesis, we propose a new algorithm SSVM-RP to obtain epsilon-optimal regularization path of structured SVM. We also propose intuitive variants of the Block-Coordinate Frank-Wolfe algorithm (BCFW) for the faster optimization of the SSVM-RP algorithm. In addition to this, we propose a principled approach to optimize the SSVM with additional box constraints using BCFW and its variants. In the end, we propose regularization path algorithm for SSVM with additional positivity/negativity constraints.In the fourth and the last part of the thesis (Appendix), we propose a novel weakly supervised discriminative algorithm for learning context specific registration metrics as a linear combination of conventional metrics. Conventional metrics can cope partially - depending on the clinical context - with tissue anatomical properties. In this work we seek to determine anatomy/tissue specific metrics as a context-specific aggregation/linear combination of known metrics. We propose a weakly supervised learning algorithm for estimating these parameters conditionally to the data semantic classes, using a weak training dataset. We show the efficacy of our approach on three highly challenging datasets in the field of medical imaging, which vary in terms of anatomical structures and image modalities.Cette thèse présente de nouvelles méthodes pour l'application de la prédiction structurée en vision numérique et en imagerie médicale.Nos nouvelles contributions suivent quatre axes majeurs.La première partie de cette thèse étudie le problème d'inférence d'ordre supérieur.Nous présentons une nouvelle famille de problèmes de minimisation d'énergie discrète, l'étiquetage parcimonieux, encourageant la parcimonie des étiquettes.C'est une extension naturelle des problèmes connus d'étiquetage de métriques aux potentiels d'ordre élevé.Nous proposons par ailleurs une généralisation du modèle Pn-Potts, le modèle Pn-Potts hiérarchique.Enfin, nous proposons un algorithme parallélisable à proposition de mouvements avec de fortes bornes multiplicatives pour l'optimisation du modèle Pn-Potts hiérarchique et l'étiquetage parcimonieux.La seconde partie de cette thèse explore le problème de classement en utilisant de l'information d'ordre élevé.Nous introduisons deux cadres différents pour l'incorporation d'information d'ordre élevé dans le problème de classement.Le premier modèle, que nous nommons SVM binaire d'ordre supérieur (HOB-SVM), optimise une borne supérieure convexe sur l'erreur 0-1 pondérée tout en incorporant de l'information d'ordre supérieur en utilisant un vecteur de charactéristiques jointes.Le classement renvoyé par HOB-SVM est obtenu en ordonnant les exemples selon la différence entre la max-marginales de l'affectation d'un exemple à la classe associée et la max-marginale de son affectation à la classe complémentaire.Le second modèle, appelé AP-SVM d'ordre supérieur (HOAP-SVM), s'inspire d'AP-SVM et de notre premier modèle, HOB-SVM.Le modèle correspond à une optimisation d'une borne supérieure sur la précision moyenne, à l'instar d'AP-SVM, qu'il généralise en permettant également l'incorporation d'information d'ordre supérieur.Nous montrons comment un optimum local du problème d'apprentissage de HOAP-SVM peut être déterminé efficacement grâce à la procédure concave-convexe.En utilisant des jeux de données standards, nous montrons empiriquement que HOAP-SVM surpasse les modèles de référence en utilisant efficacement l'information d'ordre supérieur tout en optimisant directement la fonction d'erreur appropriée.Dans la troisième partie, nous proposons un nouvel algorithme, SSVM-RP, pour obtenir un chemin de régularisation epsilon-optimal pour les SVM structurés.Nous présentons également des variantes intuitives de l'algorithme Frank-Wolfe pour l'optimisation accélérée de SSVM-RP.De surcroît, nous proposons une approche systématique d'optimisation des SSVM avec des contraintes additionnelles de boîte en utilisant BCFW et ses variantes.Enfin, nous proposons un algorithme de chemin de régularisation pour SSVM avec des contraintes additionnelles de positivité/negativité.Dans la quatrième et dernière partie de la thèse, en appendice, nous montrons comment le cadre de l'apprentissage semi-supervisé des SVM à variables latentes peut être employé pour apprendre les paramètres d'un problème complexe de recalage déformable.Nous proposons un nouvel algorithme discriminatif semi-supervisé pour apprendre des métriques de recalage spécifiques au contexte comme une combinaison linéaire des métriques conventionnelles.Selon l'application, les métriques traditionnelles sont seulement partiellement sensibles aux propriétés anatomiques des tissus.Dans ce travail, nous cherchons à déterminer des métriques spécifiques à l'anatomie et aux tissus, par agrégation linéaire de métriques connues.Nous proposons un algorithme d'apprentissage semi-supervisé pour estimer ces paramètres conditionnellement aux classes sémantiques des données, en utilisant un jeu de données faiblement annoté.Nous démontrons l'efficacité de notre approche sur trois jeux de données particulièrement difficiles dans le domaine de l'imagerie médicale, variables en terme de structures anatomiques et de modalités d'imagerie

APARCH Models Estimated by Support Vector Regression

This thesis presents a comprehensive study of asymmetric power autoregressive conditional heteroschedasticity (APARCH) models for modelling volatility in financial return data. The goal is to estimate and forecast volatility in financial data with excess kurtosis, volatility clustering and asymmetric distribution. Models based on maximum likelihood estimation (MLE) will be compared to the kernel based support vector regression (SVR). The popular Gaussian kernel and a wavelet based kernel will be used for the SVR. The methods will be tested on empirical data, including stock index prices, credit spreads and electric power prices. The results indicate that asymmetric power models are needed to capture the asseymtry in the data. Furthermore, SVR models are able to improve estimation and forecasting accuracy, compared with the APARCH models based on MLE.Masteroppgave i statistikkSTAT399MAMN-STA

Predictive control approaches to fault tolerant control of wind turbines

This thesis focuses on active fault tolerant control (AFTC) of wind turbine systems. Faults in wind turbine systems can be in the form of sensor faults, actuator faults, or component faults. These faults can occur in different locations, such as the wind speed sensor, the generator system, drive train system or pitch system. In this thesis, some AFTC schemes are proposed for wind turbine faults in the above locations. Model predictive control (MPC) is used in these schemes to design the wind turbine controller such that system constraints and dual control goals of the wind turbine are considered. In order to deal with the nonlinearity in the turbine model, MPC is combined with Takagi-Sugeno (T-S) fuzzy modelling. Different fault diagnosis methods are also proposed in different AFTC schemes to isolate or estimate wind turbine faults.The main contributions of the thesis are summarized as follows:A new effective wind speed (EWS) estimation method via least-squares support vector machines (LSSVM) is proposed. Measurements from the wind turbine rotor speed sensor and the generator speed sensor are utilized by LSSVM to estimate the EWS. Following the EWS estimation, a wind speed sensor fault isolation scheme via LSSVM is proposed.A robust predictive controller is designed to consider the EWS estimation error. This predictive controller serves as the baseline controller for the wind turbine system operating in the region below rated wind speed.T-S fuzzy MPC combining MPC and T-S fuzzy modelling is proposed to design the wind turbine controller. MPC can deal with wind turbine system constraints externally. On the other hand, T-S fuzzy modelling can approximate the nonlinear wind turbine system with a linear time varying (LTV) model such that controller design can be based on this LTV model. Therefore, the advantages of MPC and T-S fuzzy modelling are both preserved in the proposed T-S fuzzy MPC.A T-S fuzzy observer, based on online eigenvalue assignment, is proposed as the sensor fault isolation scheme for the wind turbine system. In this approach, the fuzzy observer is proposed to deal with the nonlinearity in the wind turbine system and estimate system states. Furthermore, the residual signal generated from this fuzzy observer is used to isolate the faulty sensor.A sensor fault diagnosis strategy utilizing both analytical and hardware redundancies is proposed for wind turbine systems. This approach is proposed due to the fact that in the real application scenario, both analytical and hardware redundancies of wind turbines are available for designing AFTC systems.An actuator fault estimation method based on moving horizon estimation (MHE) is proposed for wind turbine systems. The estimated fault by MHE is then compensated by a T-S fuzzy predictive controller. The fault estimation unit and the T-S fuzzy predictive controller are combined to form an AFTC scheme for wind turbine actuator faults