Meijster, Arnold

Millea, Adrian

Schomaker, Lambertus

Schutten, Marten

Wiering, Marco

English

In this paper we describe a novel extension of the support vector machine, called the deep support vector machine (DSVM). The original SVM has a single layer with kernel functions and is therefore a shallow model. The DSVM can use an arbitrary number of layers, in which lower-level layers contain support vector machines that learn to extract relevant features from the input patterns or from the extracted features of one layer below. The highest level SVM performs the actual prediction using the highest-level extracted features as inputs. The system is trained by a simple gradient ascent learning rule on a min-max formulation of the optimization problem. A two-layer DSVM is compared to the regular SVM on ten regression datasets and the results show that the DSVM outperforms the SVM

ARTS repository - University of Groningen

   University of GroningenDeep Support Vector Machines for Regression ProblemsWiering, Marco; Schutten, Marten; Millea, Adrian; Meijster, Arnold; Schomaker, LambertusPublished in: International Workshop on Advances in Regularization, Optimization, Kernel Methods, and Support VectorMachines: theory and applicationsIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionFinal author's version (accepted by publisher, after peer review)Publication date:2013Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Wiering, M., Schutten, M., Millea, A., Meijster, A., & Schomaker, L. (2013). Deep Support Vector Machinesfor Regression Problems. In International Workshop on Advances in Regularization, Optimization, KernelMethods, and Support Vector Machines: theory and applications (pp. 53-54).CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 07-06-2022Deep Support Vector Machines for Regression ProblemsM.A. Wiering, M. Schutten, A. Millea, A. Meijster, and L.R.B. SchomakerInstitute of Artificial Intelligence and Cognitive EngineeringUniversity of Groningen, the NetherlandsContact e-mail: m.a.wiering@rug.nlAbstract: In this paper we describe a novel extension of the support vector machine, calledthe deep support vector machine (DSVM). The original SVM has a single layer with kernelfunctions and is therefore a shallow model. The DSVM can use an arbitrary number of layers,in which lower-level layers contain support vector machines that learn to extract relevantfeatures from the input patterns or from the extracted features of one layer below. Thehighest level SVM performs the actual prediction using the highest-level extracted featuresas inputs. The system is trained by a simple gradient ascent learning rule on a min-maxformulation of the optimization problem. A two-layer DSVM is compared to the regular SVMon ten regression datasets and the results show that the DSVM outperforms the SVM.Keywords: Support Vector Machines, Kernel Learning, Deep Architectures1 IntroductionMachine learning algorithms are very useful for re-gression and classification problems. These algorithmslearn to extract a predictive model from a dataset ofexamples containing input vectors and target outputs.Among all machine learning algorithms, one of the mostpopular methods is the SVM. SVMs have been used formany engineering applications such as object recogni-tion, document classification, and different applicationsin bio-informatics, medicine and chemistry.Limitations of the SVM. There are two importantlimitations of the standard SVM. The first one is thatthe standard SVM only has a single adjustable layer ofmodel parameters. Instead of using such “shallow mod-els”, deep architectures are a promising alternative [4].Furthermore, SVMs use a-priori chosen kernel functionsto compute similarities between input vectors. A prob-lem is that using the best kernel function is important,but kernel functions are not very flexible.Related Work. Currently there is a lot of research inmulti-kernel learning (MKL) [1, 5]. In MKL, differentkernels are combined in a linear or non-linear way tocreate more powerful similarity functions for comparinginput vectors. However, often only few parameters areadapted in the (non-linear) combination functions. In[2], another framework for two-layer kernel machines isdescribed, but no experiments were performed in whichboth layers used non-linear kernels.Contributions. We propose the deep SVM (DSVM),a novel algorithm that uses SVMs to learn to extracthigher-level features from the input vectors, after whichthese features are given to the main SVM to do the ac-tual prediction. The whole system is trained with sim-ple gradient ascent and descent learning algorithms onthe dual objective of the main SVM. The main SVMlearns to maximize this objective, while the feature-layer SVMs learn to minimize it. Instead of adaptingfew kernel weights, we use large DSVM architectures,sometimes consisting of a hundred SVMs in the firstlayer. Still, the complexity of our DSVM scales onlylinearly with the number of SVMs compared to thestandard SVM. Furthermore, the strong regularizationpower of the main SVM prevents overfitting.2 The Deep Support Vector Machine[x]1 // .-,()*+JJJJJJJJJ666666666666666..................... f(x)[x]2 // .-,()*+HHHHHHHHHH666666666666666 S1 /.-,()*+KKKKKKKKKKK... S2/.-,()*+ M /.-,()*+ f //[x]D−1 // .-,()*+vvvvvvvvvvS3 /.-,()*+sssssssssss[x]D // .-,()*+sssssssssFig. 1: Architecture of a two-layer DSVM. In this example,the feature layer consists of three SVMs Sa.We use regression datasets: {(x1, y1), . . . , (x`, y`)},where xi are input vectors and yi are the target out-puts. The architecture of a two-layer DSVM is shownin Figure 1. First, it contains an input layer of D in-puts. Then, there are a total of d pseudo-randomlyinitialized SVMs Sa, each one learning to extract onefeature f(x)a from an input pattern x. Finally, there isthe main support vector machine M that approximatesthe target function using the extracted feature vector asinput. For computing the feature-layer representationf(x) of input vector x, we use:f(x)a =∑̀i=1(α∗i (a)− αi(a))K(xi,x) + ba,which iteratively computes each element f(x)a. In thisequation, α∗i (a) and αi(a) are SVM coefficients for SVMSa, ba is its bias, and K(·, ·) is a kernel function. Forcomputing the output of the whole system, we use:g(f(x)) =∑̀i=1(α∗i − αi)K(f(xi), f(x)) + b.Learning Algorithm. The learning algorithm adjuststhe SVM coefficients of all SVMs through a min-maxformulation of the dual objective W of the main SVM:minf(x)maxα,α∗W (f(x),α(∗)) = −ε∑̀i=1(α∗i + αi) +∑̀i=1(α∗i − αi)yi−12∑̀i,j=1(α∗i − αi)(α∗j − αj)K(f(xi), f(xj))We have developed a simple gradient ascent algorithmto train the SVMs. The method adapts the SVM co-efficients α(∗) (standing for all α∗i and αi) toward a(local) maximum of W , where λ is the learning rate:α(∗)i ← α(∗)i + λ · ∂W/∂α(∗)i . The resulting gradientascent learning rule for αi is:αi = αi + λ(−ε− yi +∑j(α∗j − αj)K(f(xi), f(xj)))We use radial basis function (RBF) kernels in both lay-ers of a two-layered DSVM. Results with other kernelswere worse. For the main SVM:K(f(xi), f(x)) = exp(−∑a(f(xi)a − f(x)a)2σm)The system constructs a new dataset for each feature-layer SVM Sa with a backpropagation-like technique formaking examples: (xi, f(xi)a − µ · δW/δf(xi)a), whereµ is some learning rate, and δW/δf(xi)a is given by:δWδf(xi)a= (α∗i − αi)∑̀j=1(α∗j − αj)f(xi)a − f(xj)aσm·K(f(xi), f(xj))The feature extracting SVMs are pseudo-randomly ini-tialized and then alternated training of the main SVMand feature layer SVMs is executed a number of epochs.The bias values are computed from the average errors.3 Experimental ResultsWe experimented with 10 regression datasets to com-pare the DSVM to an SVM, both using RBF kernels.Both methods are trained with our simple gradient as-cent learning rule, adapted to also consider penalties,e.g. for obeying the bias constraint. The first 8 datasetsare described in [3] and the other 2 datasets are takenfrom the UCI repository. The number of examples perdataset ranges from 43 to 1049, and the number of fea-tures is between 2 and 13. The datasets are split into90% trainingdata and 10% testingdata. For optimizingthe learning parameters we have used particle swarmoptimization. Finally, we used 1000 or 4000 times cross-validation with the best found parameters to computethe mean squared error and its standard error.Dataset SVM results DSVM resultsBaseball 0.02413 ± 0.00011 0.02294 ± 0.00010Boston H. 0.006838± 0.000095 0.006381 ± 0.000090Concrete . 0.00706± 0.00007 0.00621± 0.00005Electrical 0.00638 ± 0.00007 0.00641 ± 0.00007Diabetes 0.02719± 0.00026 0.02327± 0.00022Machine-CPU 0.00805± 0.00018 0.00638± 0.00012Mortgage 0.000080 ± 0.000001 0.000080 ± 0.000001Stock 0.000862± 0.000006 0.000757± 0.000005Auto-MPG 6.852 ± 0.091 6.715 ± 0.092Housing 8.71 ± 0.14 9.30 ± 0.15Tab. 1: The mean squared errors and standard errors ofthe SVM and the two-layer DSVM on 10 regression datasets.Table 1 shows the results. The results of the DSVMare significantly better for 6 datasets (p < 0.001) andworse on one. From the results we can conclude thatthe DSVM is a powerful novel machine learning algo-rithm. More research, such as adding more layers andimplementing more powerful techniques to scale up tobig datasets, can be done to discover its full potential.References[1] F.R. Bach, G.R.G. Lanckriet, and M.I. Jordan.Multiple kernel learning, conic duality, and theSMO algorithm. In Proceedings of the twenty-first international conference on Machine learning,ICML ’04, pages 6–15, 2004.[2] Francesco Dinuzzo. Kernel machines with two layersand multiple kernel learning. CoRR, 2010.[3] M. Graczyk, T. Lasota, Z. Telec, and B. Trawin-ski. Nonparametric statistical analysis of machinelearning algorithms for regression problems. InKnowledge-Based and Intelligent Information andEngineering Systems, pages 111–120. 2010.[4] G.E. Hinton and R.R. Salakhutdinov. Reducing thedimensionality of data with neural networks. Sci-ence, 313:504–507, 2006.[5] S. Sonnenburg, G. Rätsch, C. Schäfer, andB. Schölkopf. Large scale multiple kernel learn-ing. Journal of Machine Learning Research, 7:1531–1565, 2006.

Deep Support Vector Machines for Regression Problems

https://pure.rug.nl/ws/files/19535038/DSVM_extended_abstract.pdf

Deep Support Vector Machines for Regression Problems

Abstract

Similar works

Full text

Available Versions

ARTS repository - University of Groningen