Benchmarking least squares support vector machine classifiers.

In Support Vector Machines (SVMs), the solution of the classification problem is characterized by a ( convex) quadratic programming (QP) problem. In a modified version of SVMs, called Least Squares SVM classifiers (LS-SVMs), a least squares cost function is proposed so as to obtain a linear set of equations in the dual space. While the SVM classifier has a large margin interpretation, the LS-SVM formulation is related in this paper to a ridge regression approach for classification with binary targets and to Fisher's linear discriminant analysis in the feature space. Multiclass categorization problems are represented by a set of binary classifiers using different output coding schemes. While regularization is used to control the effective number of parameters of the LS-SVM classifier, the sparseness property of SVMs is lost due to the choice of the 2-norm. Sparseness can be imposed in a second stage by gradually pruning the support value spectrum and optimizing the hyperparameters during the sparse approximation procedure. In this paper, twenty public domain benchmark datasets are used to evaluate the test set performance of LS-SVM classifiers with linear, polynomial and radial basis function (RBF) kernels. Both the SVM and LS-SVM classifier with RBF kernel in combination with standard cross-validation procedures for hyperparameter selection achieve comparable test set performances. These SVM and LS-SVM performances are consistently very good when compared to a variety of methods described in the literature including decision tree based algorithms, statistical algorithms and instance based learning methods. We show on ten UCI datasets that the LS-SVM sparse approximation procedure can be successfully applied.least squares support vector machines; multiclass support vector machines; sparse approximation; discriminant-analysis; sparse approximation; learning algorithms; classification; framework; kernels; time; SISTA;

SMO-based pruning methods for sparse least squares support vector machines

Solutions of least squares support vector machines (LS-SVMs) are typically nonsparse. The sparseness is imposed by subsequently omitting data that introduce the smallest training errors and retraining the remaining data. Iterative retraining requires more intensive computations than training a single nonsparse LS-SVM. In this paper, we propose a new pruning algorithm for sparse LS-SVMs: the sequential minimal optimization (SMO) method is introduced into pruning process; in addition, instead of determining the pruning points by errors, we omit the data points that will introduce minimum changes to a dual objective function. This new criterion is computationally efficient. The effectiveness of the proposed method in terms of computational cost and classification accuracy is demonstrated by numerical experiments

Sparse least squares support vector regression for nonstationary systems

A new adaptive sparse least squares support vector regression algorithm, referred to as SLSSVR has been introduced for the adaptive modeling of nonstationary systems. Using a sliding window of recent data set of size N to track t he non-stationary characteristics of the incoming data, our adaptive model is initially formulated based on least squares support vector regression with forgetting factor (without bias term). In order to obtain a sparse model in which some parameters are exactly zeros, a l 1 penalty was applied in parameter estimation in the dual problem. Furthermore we exploit the fact that since the associated system/kernel matrix in positive definite, the dual solution of least squares support vector machine without bias term, can be solved iteratively with guaranteed convergence. Furthermore since the models between two consecutive time steps there are (N-1) shared kernels/parameters, the online solution can be obtained efficiently using coordinate descent algorithm in the form of Gauss-Seidel algorithm with minimal number of iterations. This allows a very sparse model per time step to be obtained very efficiently, avoiding expensive matrix inversion. The real stock market dataset and simulated examples have shown that the proposed approaches can lead to superior performances in comparison with the linear recursive least algorithm and a number of online non-linear approaches in terms of modelling performance and model size

A Sparse Learning Machine for Real-Time SOC Estimation of Li-ion Batteries

The state of charge (SOC) estimation of Li-ion batteries has attracted substantial interests in recent years. Kalman Filter has been widely used in real-time battery SOC estimation, however, to build a suitable dynamic battery state-space model is a key challenge, and most existing methods still use the off-line modelling approach. This paper tackles the challenge by proposing a novel sparse learning machine for real-time SOC estimation. This is achieved first by developing a new learning machine based on the traditional least squares support vector machine (LS-SVM) to capture the process dynamics of Li-ion batteries in real-time. The least squares support vector machine is the least squares version of the conventional support vector machines (SVMs) which suffers from low model sparseness. The proposed learning machine reduces the dimension of the projected high dimensional feature space with no loss of input information, leading to improved model sparsity and accuracy. To accelerate computation, mapping functions in the high feature space are selected using a fast recursive method. To further improve the model accuracy, a weighted regularization scheme and the differential evolution (DE) method are used to optimize the parameters. Then, an unscented Kalman filter (UKF) is used for real-time SOC estimation based on the proposed sparse learning machine model. Experimental results on the Federal Urban Drive Schedule (FUDS) test data reveal that the performance of the proposed algorithm is significantly enhanced, where the maximum absolute error is only one sixth of that obtained by the conventional LS-SVMs and the mean square error of the SOC estimations reaches to 10 −7 , while the proposed method is executed nearly 10 times faster than the conventional LS-SVMs

A new approach of top-down induction of decision trees for knowledge discovery

Top-down induction of decision trees is the most popular technique for classification in the field of data mining and knowledge discovery. Quinlan developed the basic induction algorithm of decision trees, ID3 (1984), and extended to C4.5 (1993). There is a lot of research work for dealing with a single attribute decision-making node (so-called the first-order decision) of decision trees. Murphy and Pazzani (1991) addressed about multiple-attribute conditions at decision-making nodes. They show that higher order decision-making generates smaller decision trees and better accuracy. However, there always exist NP-complete combinations of multiple-attribute decision-makings.;We develop a new algorithm of second-order decision-tree inductions (SODI) for nominal attributes. The induction rules of first-order decision trees are combined by \u27AND\u27 logic only, but those of SODI consist of \u27AND\u27, \u27OR\u27, and \u27OTHERWISE\u27 logics. It generates more accurate results and smaller decision trees than any first-order decision tree inductions.;Quinlan used information gains via VC-dimension (Vapnik-Chevonenkis; Vapnik, 1995) for clustering the experimental values for each numerical attribute. However, many researchers have discovered the weakness of the use of VC-dim analysis. Bennett (1997) sophistically applies support vector machines (SVM) to decision tree induction. We suggest a heuristic algorithm (SVMM; SVM for Multi-category) that combines a TDIDT scheme with SVM. In this thesis it will be also addressed how to solve multiclass classification problems.;Our final goal for this thesis is IDSS (Induction of Decision Trees using SODI and SVMM). We will address how to combine SODI and SVMM for the construction of top-down induction of decision trees in order to minimize the generalized penalty cost

Design and implementation of a soft computing-based controller for a complex mechanical system

Soft-Computing basierende Regler beinhalten Algorithmen, die im Bereich des Maschinellen Lernens einzuordnen sind. Diese Regler sind in der Lage eine geeignete Steuerungsstrategie durch direkte Interaktion mit einer dynamischen Regelstrecke zu entwerfen. Sowohl klassische als auch moderne Reglerentwurfsmethoden hangen von der Genauigkeit des verwendeten dynamischen Systemmodells ab, was insbesondere bei steigender Komplexitat des Systems und auftretenden Modellunsicherheiten nicht mehr uneingeschrankt gewahrleistet werden kann. Die Ziele von Soft- Computing basierenden Reglern sind die Verbesserung der Gute des Regelverhaltens und eine geeignete Anpassung der Regler ohne eine mathematische Modellbildung auf Grundlage von physikalischen Gesetzen. Im Rahmen dieser Arbeit werden funf Algorithmen zur Modellbildung und Regelung dynamischer Systeme untersucht, welche auf dem Mehrschichten-Perzeptron-Netzwerk (Multi-Layer Perceptron network, MLP), auf der Methode der Support Vector Machine (SVM), der Gau-Prozesse, der radialen Basisfunktionen (Radial Basis Functions, RBF) sowie der Fuzzy-Inferenz-Systeme basieren. Im Anschluss an die Darstellung der zugrunde liegenden mathematischen Zusammenhange dieser Methoden sowie deren Hauptanwendungsfelder im Bereich der Modellbildung und Regelung dynamischer Systeme wird eine systematische Evaluierung der funf Methoden diskutiert. Anhand der Verwendung quantitativer Gutekennziern werden diese Methoden fur die Verwendung in der Modellbildung und Regelung dynamischer Systeme vergleichbar gegenubergestellt. Basierend auf den Ergebnissen der Evaluierung wird der SVM-basierte Algorithmus als Kernalgorithmus des Soft-Computing basierenden Reglers verwendet. Der vorgestellte Regler besteht aus zwei Hauptteilen, wobei der erste Teil aus einer Modellfunktion der dynamischen Regelstrecke und einem SVM-basierten Beobachter besteht, und der zweite Teil basierend auf dem Systemmodell eine geeignete Regelstrategie generiert. Die Verikation des SVM-basierten Regleralgorithmus erfolgt anhand eines FEM-Modells eines dynamischen elastischen Balken bzw. einseitig eingespannten elastischen Balkens. Dieses Modell kann z. B. als Ersatzmodell fur das mechanische Verhalten eines exiblen Roboterarms oder einer Flugzeugtrag ache verwendet werden. Der Hauptteil der Modellfunktion besteht aus einem automatischen Systemidentikationsalgorithmus, der auch die Integration eines systematischen Modellbildungsansatzes fur dynamische Systeme ermoglicht.Die Ergebnisse des SVM-basierten Beobachter zeigen ahnliches Verhalten zum Kalman- Bucy Beobachter. Auch die Sensitivitatsanalyse der Parameter zeigt eine bessere Gute der SVM-basierten Beobachter im Vergleich mit den Kalman-Bucy Beobachtern. Im Anschluss wird der SVM-basierte Regler zur Schwingungsregelung des Kragtragers verwendet. Hierbei werden vergleichbare Ergebnisse zum LQR-Regler erzielt. Eine experimentelle Validierung des SVM basierten Reglers erfolgt an Versuchsst anden eines elastischen Biegebalkens sowie eines invertierten Biegebalkens. Die Zustandsbeobachtung fuhrt zu vergleichbaren Ergebnissen verglichen mit einem Kalman-Bucy Beobachter. Auch die Modellbildung des elastischen Balkens fuhrt zu guten Ubereinstimmungen. Die Regelgute des Soft-Computing basierenden Reglers wurde am Versuchsstand des invertierten Biegebalkens experimentell erprobt. Es wird deutlich, dass Ergebnisse im Rahmen der erforderlichen Vorgaben erzielt werden konnen.The focus of this thesis is to obtain a soft computing-based controller for complex mechanical system. soft computing based controllers are based on machine learning algorithm that able to develop suitable control strategies by direct interaction with targeted dynamic systems. Classical and modern control design methods depend on the accuracy of the system dynamic model which cannot be achieved due to the dynamic system complexity and modeling uncertainties. A soft computing-based controller aims to improve the performance of the close loop system and to give the controller adaptation ability as well as to reduce the need for mathematical modeling based on physical laws. In this work ve dierent softcomputing algorithms used in the eld of modeling and controlling dynamic systems are investigated.These algorithms are Multi-Layer Perceptron(MLP) network, Support Vector Machine (SVM),Gaussian process, Radial Basis Function (RBF), and Fuzzy Inference System (FIS). The basic mathematical description of each algorithm is given. Additionally, the most recent applications in modeling and controlling of dynamic system are summarized. A systematic evaluation of the ve algorithms is proposed. The goal of the evaluation is to provide quantitative measure of the performance of soft computing algorithms when used in modeling and controlling a dynamic system. Based on the evaluation, the SVM algorithm is selected as the core learning algorithm for the soft computing based controller. The controller has two main units. The rst unit has two functions of modeling dynamic system and obtaining a SVM-based observer. The second unit is in charge of generating suitable control strategy based on the dynamic model obtained. The verication of the controller using SVM algorithm is done using an elastic cantilever beam modeled using Finite Element Method (FEM). An elastic cantilever beam can be considered as a representation of exible single-link manipulator or aircraft wing. In the core of the modeling unit, an automatic system identication algorithm which allows a systematic modeling approach of dynamic systems is implemented. The results show that the system dynamic model using SVM algorithm is accurate with respect to the FEM model. As for the SVM-based observer the results show that it has good estimation in comparison with to dierent Kalman-Bucy observers. The sensitivity to parameters variations analysis shows that the SVM-based observer has better performance than Kalman-Bucy observer. The SVM based controller is used to control the vibration of the cantilever beam; the results show that the model reference controller using SVM has a similar performance to LQR controller. The validation of the controller using SVM algorithm is carried out using the elastic cantilever beam test rig and the inverted cantilever beam test rig. The states estimation using SVM-based observer of the elastic cantilever beam test rig is successful and accurate compared to a Kalman-Bucy observer. Modeling of the elastic cantilever beam using the SVM algorithm shows good accuracy. The performance of controller is tested on the inverted cantilever beam test rig. The results show that required performance objective can be realized using this control strategy

Experimental Study on 164 Algorithms Available in Software Tools for Solving Standard Non-Linear Regression Problems

In the specialized literature, researchers can find a large number of proposals for solving regression problems that come from different research areas. However, researchers tend to use only proposals from the area in which they are experts. This paper analyses the performance of a large number of the available regression algorithms from some of the most known and widely used software tools in order to help non-expert users from other areas to properly solve their own regression problems and to help specialized researchers developing well-founded future proposals by properly comparing and identifying algorithms that will enable them to focus on significant further developments. To sum up, we have analyzed 164 algorithms that come from 14 main different families available in 6 software tools (Neural Networks, Support Vector Machines, Regression Trees, Rule-Based Methods, Stacking, Random Forests, Model trees, Generalized Linear Models, Nearest Neighbor methods, Partial Least Squares and Principal Component Regression, Multivariate Adaptive Regression Splines, Bagging, Boosting, and other methods) over 52 datasets. A new measure has also been proposed to show the goodness of each algorithm with respect to the others. Finally, a statistical analysis by non-parametric tests has been carried out over all the algorithms and on the best 30 algorithms, both with and without bagging. Results show that the algorithms from Random Forest, Model Tree and Support Vector Machine families get the best positions in the rankings obtained by the statistical tests when bagging is not considered. In addition, the use of bagging techniques significantly improves the performance of the algorithms without excessive increase in computational times.This work was supported in part by the University of Córdoba under the project PPG2019-UCOSOCIAL-03, and in part by the Spanish Ministry of Science, Innovation and Universities under Grant TIN2015- 68454-R and Grant TIN2017-89517-P

Input variable selection methods for construction of interpretable regression models

Large data sets are collected and analyzed in a variety of research problems. Modern computers allow to measure ever increasing numbers of samples and variables. Automated methods are required for the analysis, since traditional manual approaches are impractical due to the growing amount of data. In the present thesis, numerous computational methods that are based on observed data with subject to modelling assumptions are presented for producing useful knowledge from the data generating system. Input variable selection methods in both linear and nonlinear function approximation problems are proposed. Variable selection has gained more and more attention in many applications, because it assists in interpretation of the underlying phenomenon. The selected variables highlight the most relevant characteristics of the problem. In addition, the rejection of irrelevant inputs may reduce the training time and improve the prediction accuracy of the model. Linear models play an important role in data analysis, since they are computationally efficient and they form the basis for many more complicated models. In this work, the estimation of several response variables simultaneously using the linear combinations of the same subset of inputs is especially considered. Input selection methods that are originally designed for a single response variable are extended to the case of multiple responses. The assumption of linearity is not, however, adequate in all problems. Hence, artificial neural networks are applied in the modeling of unknown nonlinear dependencies between the inputs and the response. The first set of methods includes efficient stepwise selection strategies that assess usefulness of the inputs in the model. Alternatively, the problem of input selection is formulated as an optimization problem. An objective function is minimized with respect to sparsity constraints that encourage selection of the inputs. The trade-off between the prediction accuracy and the number of input variables is adjusted by continuous-valued sparsity parameters. Results from extensive experiments on both simulated functions and real benchmark data sets are reported. In comparisons with existing variable selection strategies, the proposed methods typically improve the results either by reducing the prediction error or decreasing the number of selected inputs or with respect to both of the previous criteria. The constructed sparse models are also found to produce more accurate predictions than the models including all the input variables

Data Mining

Data mining is a branch of computer science that is used to automatically extract meaningful, useful knowledge and previously unknown, hidden, interesting patterns from a large amount of data to support the decision-making process. This book presents recent theoretical and practical advances in the field of data mining. It discusses a number of data mining methods, including classification, clustering, and association rule mining. This book brings together many different successful data mining studies in various areas such as health, banking, education, software engineering, animal science, and the environment