This is an electronic version of the paper presented at the 18th European Symposium on Artificial Neural Networks, held in Bruges on 2010Least Squares Support Vector Machines (LS-SVMs) were
proposed by replacing the inequality constraints inherent to L1-SVMs with
equality constraints. So far this idea has only been suggested for a least
squares (L2) loss. We describe how this can also be done for the sumof-slacks
(L1) loss, yielding a new classifier (Least 1-Norm SVMs) which
gives similar models in terms of complexity and accuracy and that may
also be more robust than LS-SVMs with respect to outliers.With partial support of Spain’s TIN 2007–66862 project and Cátedra IIC en Modelado y
Predicción. The first author is kindly supported by FPU-MICINN grant reference AP2007–
00142

Dorronsoro, José R.

López Lázaro, Jorge

Dorronsoro Ibero, José Ramón

Biblos-e Archivo

Least 1-Norm SVMs: a New SVM Variantbetween Standard and LS-SVMsJorge Lo´pez and Jose´ R. Dorronsoro ∗Universidad Auto´noma de MadridDepartamento de Ingenier´ıa Informa´tica andInstituto de Ingenier´ıa del ConocimientoC/ Francisco Toma´s y Valiente 11, 28049 Madrid, SpainAbstract. Least Squares Support Vector Machines (LS-SVMs) wereproposed by replacing the inequality constraints inherent to L1-SVMs withequality constraints. So far this idea has only been suggested for a leastsquares (L2) loss. We describe how this can also be done for the sum-of-slacks (L1) loss, yielding a new classifier (Least 1-Norm SVMs) whichgives similar models in terms of complexity and accuracy and that mayalso be more robust than LS-SVMs with respect to outliers.1 IntroductionAssuming a binary classiﬁcation context, we have a sample of N preclassiﬁedpatterns {Xi, yi} , i = 1, . . . , N , where the outputs yi ∈ {+1,−1}. If we furtherassume linear inseparability and consider slack variables to allow for misclassiﬁ-cations, the primal of an LS-SVM [1] isminW,b,ξ12‖W‖2 + C2∑iξ2i s.t. yi (W · Φ(Xi) + b) = 1− ξi ∀i, (1)where · denotes inner product, and Φ (Xi) is the image of Xi in the feature spacewith feature map Φ (·). The corresponding dual isminα12∑i∑jαiαjyiyjK˜ij −∑iαi s.t.∑iαiyi = 0, (2)with the modiﬁed kernel K˜ij = k (Xi,Xj) + δij/C, δij standing for Kronecker’sdelta symbol and k (Xi,Xj) = Φ (Xi) · Φ(Xj) the original kernel.LS-SVMs were originally derived in [1] from the so-called L1-SVMs [2], whoseprimal changes (1) in three aspects: 1) the objective function uses the L1 lossC∑i ξi instead of the L2 loss, 2) the equality constraints become inequalityones, and 3) there is the additional requirement that ξi ≥ 0. In turn, L2-SVMs,also described in [2], lie somewhere in between, since their primal is identical to(1), but with the equality constraints still transformed into inequality ones.To our knowledge, there is no current classiﬁer that combines equality con-straints with the L1 loss. It is desirable to ﬁll this gap mainly because of two∗With partial support of Spain’s TIN 2007–66862 project and Ca´tedra IIC en Modelado yPrediccio´n. The first author is kindly supported by FPU-MICINN grant reference AP2007–00142.135ESANN 2010 proceedings, European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning.  Bruges (Belgium), 28-30 April 2010, d-side publi., ISBN 2-930307-10-2.Squared Slacks SlacksInequality constraints L2-SVMs L1-SVMsEquality constraints LS-SVMs ?Table 1: Types of SVMs according to how slacks and constraints are treated.facts: 1) in practice L1-SVMs and the L1 loss have become the standard, 2) theinﬂuence of a given pattern (i.e. the value of its coeﬃcient αi) in the model isnot bounded when using the L2 loss, so L2 and LS-SVMs are more sensitive tooutliers than L1-SVMs.The central idea of this work is to simplify L1-SVMs similarly to LS-SVMs,but keeping the L1 loss, giving rise to the so-called Least 1-Norm SVMs, whichﬁll the gap above and are expected to preserve the robustness to outliers. Therest of the paper is organized as follows: in Section 2 we give the primal anddual of Least 1-Norm SVMs and discuss brieﬂy their KKT optimality conditions.Section 3 explains how the popular SMO algorithm can be adapted to solve theLeast 1-Norm dual. Section 4 reports some experiments that illustrate how theycan be more robust to outliers than LS-SVMs while being as accurate as them,and discusses the varied convergence speeds observed. Finally, Section 5 givespointers to future possible extensions.2 Least 1-Norm SVMsIn order to simplify the L1-SVM primal, one may think that it suﬃces to forceequality constraints yi (W · Φ(Xi) + b) = 1 − ξi, while keeping the inherentrequirement ξi ≥ 0. However, this is not correct because it implies that slacksare only allowed in one direction, something which is obviously not convenient.Therefore, we propose to remove the constraints ξi ≥ 0 and minimize the 1-Normof the slack vector, which gives the Least 1-Norm SVM primalminW,b,ξ12‖W‖2 + C∑i|ξi| s.t. yi (W · Φ(Xi) + b) = 1− ξi ∀i. (3)Now we use the cast of 1-Norm problems as Linear Programming problems[3, p. 294]: minimizing (3) can be reformulated asminW,b,t12‖W‖2 + C∑iti s.t. − ti ≤ 1− yi (W · Φ(Xi) + b) ≤ ti ∀i, (4)Note that (4) transforms the desired equalities of (3) into inequalities, butotherwise the objective function is not diﬀerentiable. Using standard Lagrangiantheory with (4) and denoting βi (γi) as the multipliers associated with −ti (+ti)we obtain the following dual, where αi = γi − βi:minα12∑i∑jαiαjyiyjKij −∑iαi s.t.∑iαiyi = 0, −C ≤ αi ≤ C ∀i, (5)136ESANN 2010 proceedings, European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning.  Bruges (Belgium), 28-30 April 2010, d-side publi., ISBN 2-930307-10-2.which happens to be identical to the L1-SVM dual but with the lower bound−C instead of 0, so that negative values are allowed, as in LS-SVMs.Since all the formulations above are convex with aﬃne constraints, the KKToptimality conditions are necessary and suﬃcient for optimality [3]. The KKTconditions for (4) are analogous to the well-known ones for L1-SVMs, substitut-ing just the lower bound −C for 0, which yields:yi (W · Φ(Xi) + b) = 1 ∀i | − C < αi < C,yi (W · Φ(Xi) + b) ≤ 1 ∀i | αi = C,yi (W · Φ(Xi) + b) ≥ 1 ∀i | αi = −C, (6)together with the dual constraints W =∑i αiyiΦ(Xi) and∑i αiyi = 0. Theseare common to LS-SVMs, whose only primal KKT condition [1] isyi (W · Φ(Xi) + b) = 1− αi/C ∀i, (7)which shows why LS-SVMs are very sensible to outliers: outliers are character-ized by a large |ξi|, which in view of (7) and (1) implies a large |αi|. On theother hand, in Least 1-Norm SVMs this inﬂuence is limited because |αi| ≤ C.It also shows another drawback of LS-SVMs: they are not sparse becauseαi = Cξi ∀i, so a pattern takes part in the model whenever ξi = 0, which isalmost certain to happen. Observe that this is also the case for Least 1-NormSVMs, since αi = 0 implies yi (W · Φ(Xi) + b) is exactly 1, so they are not likelyto be sparse either. L1-SVMs are indeed sparse because, instead of −C, patternswith yi (W · Φ(Xi) + b) > 1 are assigned αi = 0.3 Least 1-Norm SMOWe will adapt SMO for Least 1-Norm SVMs basing on a maximum gain view-point (for more details see [4]). In general, SMO performs updates of theform W ′ = W + δLyLXL + δUyUXU . The constraint∑αiyi = 0 impliesδUyU = −δLyL and the updates become W ′ = W + δyL (XL −XU ), wherewe write δ = δL and, hence, δU = −yUyLδ. As a consequence, the multiplierupdates are α′L = αL + δ, α′U = αU −yUyLδ and α′j = αj for other j. Therefore,denoting the dual function in (5) as D (α), D (α′) can be written asD (α′) = D (α)− (ΔU,L)2‖ZL,U‖2 ,where we write ΔU,L = W · (XU −XL) − (yU − yL) and ZL,U = XL − XU .Ignoring the denominator, we can approximately maximize the gain in D (α′) bychoosing L = argminj {W ·Xj − yj} and U = argmaxj {W ·Xj − yj}, so thatthe violation extent ΔU,L is largest. Writing Δ = ΔU,L and λ′ = Δ/‖ZU,L‖2, wethen have Δ > 0, λ′ > 0, δ = yLλ′ and the α updates become α′L = αL + yLλ′,α′U = αU − yUλ′. Thus, α′L or α′U will decrease if yL = −1 or yU = 1, whichrequires the corresponding αL and αU to be greater than −C. In turn, they will137ESANN 2010 proceedings, European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning.  Bruges (Belgium), 28-30 April 2010, d-side publi., ISBN 2-930307-10-2.increase if yL = 1 or yU = −1, which requires the corresponding αL and αU tobe less than C. Hence, we must replace the previous L,U choices withL = argminj{W ·Xj − yj : j ∈ IL} , U = argmaxj{W ·Xj − yj : j ∈ IU} , (8)where we use the notations IU = {i : (yi = 1, αi > −C) ∨ (yi = −1, αi < C)}and IL = {i : (yi = 1, αi < C) ∨ (yi = −1, αi > −C)}. Moreover, to make surethat α′L and α′U remain then in the interval [−C,C], we may have to clip λ′ withλ′ = min {λ′, C − yLαL, C + yUαU} . (9)4 Numerical ExperimentsIn this section we will illustrate empirically how the Least 1-Norm SVM maybe more robust to outliers than its LS-SVM counterpart, as well as its goodgeneralization properties. The training algorithm is SMO; the Least 1-Normvariant explained above and the LS-SVM version in [5]. The stopping criterionis ﬁnal KKT violation, speciﬁcally when it is less than  = 10−3. For LS-SVMsthis meansmaxi{W˜ · Φ˜ (Xi)− yi}−mini{W˜ · Φ˜ (Xi)− yi}≤ , (10)where the tilde indicates that we use the modiﬁed kernel k˜ as in (2). For Least1-Norm SVMs, it meansmaxIU{W · Φ(Xi)− yi} −minIL {W · Φ(Xi)− yi} ≤ . (11)The derivation of these KKT based criteria is given in [6] for LS-SVMs and[7] for L1-SVMs. Firstly, to show generalization we take 4 datasets from [8] with100 training–test splits each. We compare the performance of Least 1-Normand LS-SVMs. We use the RBF kernel k (Xi,Xj) = exp(−‖Xi −Xj‖2/σ). Thevalues for the hyperparameters C and σ are sought with a grid in the logarithmicrange [0, 2] for C and [0, 4] for σ. Each point of the grid is evaluated with a 10-times-10-fold cross-validation over the whole dataset. We report in Table 2 theaccuracy and number of support vectors obtained in the ﬁnal models, as well asthe number of iterations needed by the corresponding SMO version to stop.LS Least 1-Norm% err. #SV #It. % err. #SV #It.Titanic 22.4±1.0 150.0±0.0 390.1±14.7 22.4±1.0 71.4±9.8 53.7±6.1Heart 15.6±3.2 169.9±0.3 261.3±7.8 15.6±3.5 169.9±0.3 120.1±14.7Cancer 25.7±4.5 199.8±0.5 424.6±7.9 25.9±4.5 195.9±2.3 3510.7±701.9German 23.3±2.1 699.4±0.8 3329.8±38.0 23.3±2.2 699.9±0.3 16167.4±1419.3Table 2: Average accuracies, number of support vectors and number of iterationsobtained by a Least 1-Norm SVM and an LS-SVM.138ESANN 2010 proceedings, European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning.  Bruges (Belgium), 28-30 April 2010, d-side publi., ISBN 2-930307-10-2.0 5 10−2024xy(a) LS−SVM without outliers, C = 1−1010 5 10−2024xy(b) Least 1−Norm SVM without outliers, C = 1−1010 5 10−2024xy(c) LS−SVM with outliers, C = 1−1010 5 10−2024xy(d) Least 1−Norm SVM with outliers, C = 1−101Fig. 1: Contours of function W ·Φ(X)+ b for a toy problem trained with an LS-SVM (left) and a Least 1-Norm SVM (right). Top: original problem. Bottom:modiﬁed problem with one outlier for each class.It can be seen how the accuracies obtained are similar for both kinds of SVMand also similar to the ones reported in [8] for an L1-SVM. Regarding the numberof support vectors, as expected, none is sparse, except the Least 1-Norm SVMfor dataset titanic, which we think is due to the existence of identical pointswith diﬀerent tags. Finally, concerning the number of iterations, it is somewhatpuzzling, sometimes the LS-SVM is remarkably faster and sometimes the Least1-Norm SVM is. This of course depends on the hyperparameters chosen, but itis not clear what is the exact inﬂuence of them. Care must also be taken since(10) and (11), though formally similar, may require quite diﬀerent number ofiterations since the W vectors are diﬀerent. Further study is clearly needed tobetter characterize what the convergence speed will be for each case.Secondly, to show robustness we use the toy bidimensional problem depictedin 1, where 20 patterns belong to each class. The positive class’ patterns aredrawn from a normal distribution with mean (0, 0), whereas the negative classhas a mean of (5, 2). In both cases the covariance matrix is the unit one. In thetop part of the ﬁgure we train an LS-SVM (a) and a Least 1-Norm SVM (b)with this training set, which is linearly separable, with C = 1 and no speciﬁckernel (just the inner product). Note that the ﬁnal hyperplanes are very similarand the “support”hyperplanes traverse their corresponding cloud of points.In the bottom part of the ﬁgure, we introduce two outliers by switching theclass labels of two points, so that the classes are no longer linearly separable,training again the LS-SVM (c) and the Least 1-Norm SVM (d). Observe thatthe ﬁnal LS-SVM hyperplane has remarkably changed its orientation because ofthe outliers’ inﬂuence, whereas the Least 1-Norm one changes quite less because139ESANN 2010 proceedings, European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning.  Bruges (Belgium), 28-30 April 2010, d-side publi., ISBN 2-930307-10-2.their inﬂuence is limited.5 Conclusions and further workIn this work we have presented Least 1-Norm SVMs, a new SVM classiﬁer. AsLS-SVMs did with L1-SVMs, they are derived by substituting inequality forequality constraints in the primal. The arising dual is almost identical to theL1 one, with box constraints [−C,C] in lieu of [0, C]. This implies that theoutliers’ inﬂuence is also limited, but sparsity is lost because now the points forwhich yiW · Φ(Xi) > 1 are assigned an αi = −C instead of being zero. Wehave also seen how it can be trained with an adaptation of the well-known SMOalgorithm, giving models with similar test accuracies. Which particular SVMvariant converges faster seems to be problem and parameter dependent.As a possible future extension, the training phase for Least 1-Norm SVMscan be accelerated by making use of the 2nd order variant of the SMO algorithmas was done for L1-SVMs in [7]. This method has been shown to not alwaysaccelerate LS-SVM training [6]. As mentioned above, the convergence propertiesof SMO for Least 1-Norm SVMs will be further studied.References[1] J. A. K. Suykens and J. Vandewalle. Least Squares Support Vector MachineClassiﬁers. Neural Processing Letters, 9(3):293–300, 1999.[2] V. Vapnik. The Nature of Statistical Learning Theory. Springer-Verlag, NewYork, 1995.[3] S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge UniversityPress, 2004.[4] J. Lo´pez, A´. Barbero, and J. R. Dorronsoro. On the Equivalence of the SMOand MDM Algorithms for SVM Training. In Lecture Notes in ComputerScience: Machine Learning and Knowledge Discovery in Databases, volume5211, pages 288–300. Springer, 2008.[5] S. S. Keerthi and S. K. Shevade. SMO Algorithm for Least-Squares SVMFormulations. Neural Computation, 15(2):487–507, 2003.[6] J. Lo´pez and J. A. K. Suykens. First and Second Order SMO Algorithmsfor Large Scale LS-SVM training. Technical Report 09-179, Katholieke Uni-versiteit Leuven, 2009.[7] R. E. Fan, P. H. Chen, and C. J. Lin. Working Set Selection using Sec-ond Order Information for Training Support Vector Machines. Journal ofMachine Learning Research, 6:1889–1918, 2005.[8] G. Ra¨tsch. Benchmark Repository, 2000. Datasets available at http://ida.first.fhg.de/projects/bench/benchmarks.htm.140ESANN 2010 proceedings, European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning.  Bruges (Belgium), 28-30 April 2010, d-side publi., ISBN 2-930307-10-2.

Least 1-Norm SVMs: a new SVM variant between standard and LS-SVMs

https://repositorio.uam.es/bitstream/handle/10486/667700/least_lopez_ESANN_2010.pdf?sequence=1

Least 1-Norm SVMs: a new SVM variant between standard and LS-SVMs

Abstract

Similar works

Full text

Available Versions

Biblos-e Archivo