In this paper, a generalization of support vector machines is explored where it is considered that input vectors have different ℓp norms for each class. It is proved that the optimization problem for binary classification by using the maximal margin principle with ℓp and ℓq norms only depends on the ℓp norm if 1 ≤ p ≤ q. Furthermore, the selection of a different bias in the classifier function is a consequence of the ℓq norm in this approach. Some commentaries on the most commonly used approaches of SVM are also given as particular cases

Franco Martín, Luis

González Abril, Luis

Ortega Ramírez, Juan Antonio

Velasco Morente, Francisco

English

idUS. Depósito de Investigación Universidad de Sevilla

Computers and Mathematics with Applications 61 (2011) 2874–2878Contents lists available at ScienceDirectComputers and Mathematics with Applicationsjournal homepage: www.elsevier.com/locate/camwaSupport vector machines for classification of input vectors withdifferent metricsL. Gonzalez-Abril a,∗, F. Velasco a, J.A. Ortega b, L. Franco aa Applied Economics I Department, Seville University, 41018 Seville, Spainb Computer Languages and Systems Department, Seville University, 41012 Seville, Spaina r t i c l e i n f oArticle history:Received 8 January 2010Received in revised form 15 March 2011Accepted 21 March 2011Keywords:Pattern recognitionLearning machineMaximal margin principleℓp norma b s t r a c tIn this paper, a generalization of support vectormachines is exploredwhere it is consideredthat input vectors have different ℓp norms for each class. It is proved that the optimizationproblem for binary classification by using the maximal margin principle with ℓp and ℓqnorms only depends on the ℓp norm if 1 ≤ p ≤ q. Furthermore, the selection of a differentbias in the classifier function is a consequence of the ℓq norm in this approach. Somecommentaries on the most commonly used approaches of SVM are also given as particularcases.© 2011 Elsevier Ltd. All rights reserved.1. IntroductionSupport vector machines (SVMs) are learning machines which implement the structural risk-minimization inductiveprinciple to obtain good generalization on a limited number of learning patterns [1]. This theory was developed on the basisof a separable binary classification problem where the output scale is determined such that outputs for the support vectorsare ±1. The optimization criterion is the width of the margin between the positive and negative examples, since an SVMwith a large margin separating two classes has a small VC dimension which provides a good generalization performance, ashas been demonstrated in some applications [2].Although the ℓp norm has been explored in SVMs [3,4], the norm usually used is ℓ2 or ℓ∞ norm and is always the samenorm for both positive andnegative examples. In this paper, it is considered that norm for the positive andnegative examplesare ℓp and ℓq norms, where1 p, q ≥ 1, respectively. It is worth noting that some real-world examples that justify usingmultiple metrics and theoretical results of this approach have recently been considered in other areas of research [5–9].Nevertheless, to the best of our knowledge, there are not existing researches that directly address the problem of patternrecognition using this approach.The remainder of this paper is arranged as follows: Section 2 presents a new SVM to solve linearly separable binaryclassification problems with a different metric for each class. Some particular cases are obtained and comments aremade on this approach in Section 3, and its extensions to solve multiclassification and nonlinear separable problemsare given in Section 4. Empirical testing is carried out using different biases in Section 5. Finally, some conclusions aredrawn.∗ Corresponding author. Tel.: +34 954554345; fax: +34 954551636.E-mail addresses: luisgon@us.es (L. Gonzalez-Abril), velasco@us.es (F. Velasco), ortega@lsi.us.es (J.A. Ortega), lfranco@us.es (L. Franco).1 Note that ℓp is not a norm if p < 1.0898-1221/$ – see front matter© 2011 Elsevier Ltd. All rights reserved.doi:10.1016/j.camwa.2011.03.071L. Gonzalez-Abril et al. / Computers and Mathematics with Applications 61 (2011) 2874–2878 28752. ℓp-SVM-ℓq approachLet Z = {(x1, y1), . . . , (xn, yn)} be a training set, with2 xi = (xi1, . . . , xid)′ ∈ Rd as the input space, yi ∈ Y = {θ1, θ2} ={+1, −1} the output space, and zi = (xi, yi) for i = 1, . . . , n. LetZ+ andZ− be the patterns belonging to the classes labelledas +1 and −1, respectively.Let us first consider the linearly separable case, that is, a w ∈ Rd (no null), b ∈ R, xi0 ∈ Z+ and xj0 ∈ Z− exist suchthat w′ x + b ≥ w′ xi0 + b = 1 if x ∈ Z+, and w′ x + b ≤ w′ xj0 + b = −1 if x ∈ Z−. Hence, both inequalities can bewritten as yiw′ xi + b≥ 1 for any i = 1, . . . , n. The best pair is sought among all pairs (w, b) by following the criterionof maximization of the margin [1].Given b∗ with −1 ≤ b∗ ≤ 1, let us consider the hyperplane πb∗ : w′ x + b − b∗ = 0 and suppose, without any loss ofgenerality, that norms in the region πb∗ > 0 and πb∗ < 0 are the ℓp and ℓq norms, respectively, with 1 ≤ p ≤ q. Thus, givenxi ∈ Z+ and xj ∈ Z−, the p-distance between xi and πb∗ , that is dp(xi, πb∗) = minx∈πb∗ dp(xi, x), is (see in [10,3,11]):dp(xi, πb∗) =w′ xi + b − b∗‖w‖p1=w′ xi + b − b∗‖w‖p1≥1 − b∗‖w‖p1= dp(xi0, πb∗) (1)and the q-distance between xj and πb∗ is as follows:dq(xj, πb∗) =w′ xj + b − b∗‖w‖q1=−w′ xj + b − b∗‖w‖q1≥1 + b∗‖w‖q1= dq(xj0, πb∗) (2)on the condition that p1 and q1 are the conjugate exponents of p and q, respectively, that is,3 1p +1p1= 1, 1q +1q1= 1, and‖w‖p =∑Mi=1 |wi|p 1p, ‖w‖∞ = maxi |wi| are the p-norm and the ∞-norm of the vectorw, respectively.By defining the p-distance between a set A and a hyperplane πb∗ , denoted by dp(A, πb∗), as dp(A, πb∗) = infdp(x, πb∗), for all x ∈ A, then by using (1) and (2) (the lowest bound is always attained in Z+ and Z−):dp(Z+, πb∗) =1 − b∗‖w‖p1and dq(Z−, πb∗) =1 + b∗‖w‖q1.Hence, the pq-margin between Z+ and Z− with respect to the hyperplane πb∗ can be defined as dp(Z+, πb∗) +dq(Z−, πb∗). Therefore, the hyperplane πb∗ with the largest pq-margin in Z can be obtained by solving the problemmaxdp(Z+, πb∗) + dq(Z−, πb∗), which can be written as follows:maxw,b,b∗1 − b∗‖w‖p1+1 + b∗‖w‖q1s. t. yiw′ xi + b≥ 1, −1 ≤ b∗ ≤ 1, ∀i = 1, . . . , n.(3)It is worth noting that the constraints do not depend on p and q. This optimization problem can be simplified since1 ≤ p ≤ q ⇒ p1 ≥ q1 ⇒ ‖w‖p1 ≤ ‖w‖q1 for any w ∈ Rd. Hence,1 − b∗‖w‖p1+1 + b∗‖w‖q1≤1 − b∗‖w‖p1+1 + b∗‖w‖p1≤2‖w‖p1for any −1 ≤ b∗ ≤ 1, and as this upper bound is attained by taking b∗ = −1; therefore, the optimization problem (3) isequivalent to maxw,b 2‖w‖p1, which can be formulated as a pth-order programming problem [10]:minw,b12‖w‖p1p1s. t. yiw′ xi + b≥ 1, ∀i = 1, . . . , n.(4)The objective function 12 ‖w‖p1p1 is a convex function for any p1 ≥ 1 and the constraints are linear; therefore, theoptimization problem (4) has a unique solution. It is important to note that the problem (4) does not depend on the ℓqnorm (q ≥ p) and regardless of whether the ℓp norm is considered in positive or negative examples.Therefore, a binary linear classifier, f(w,b)(x) = w′ x + b − 1, is found with w = (w1, . . . , wd)′ ∈ Rd and b ∈ R(b∗ = −1), and where outputs are obtained as h(w,b)(x) = sign(f(w,b)(x)), that is, +1 (θ1) if f(w,b)(x) ≥ 0, and −1 (θ2)otherwise. Nevertheless, a new bias, different to b − 1 (that is b∗ = 1) and b (that is b∗ = 0), must be considered since in2 Notation: vectors are denoted in bold, the transposed vector of x is denoted by x′ . Sets and real numbers are denoted in capitals and lower-case letters,respectively.3 If p = 1, then p1 = ∞.2876 L. Gonzalez-Abril et al. / Computers and Mathematics with Applications 61 (2011) 2874–2878these cases dq(Z−, π−1) = 0 and dq(Z−, π0) = 1‖w‖q1≤1‖w‖p1= dp(Z+, π0), that is, the vectors of Z− are closer thanthe vectors of Z+ to the hyperplanes π−1 and π0. Thus, a natural selection of the b∗ is to choose it such that the equalitydq(Z−, πb∗) = dp(Z+, πb∗) holds, and thus:1 − b∗0‖w‖p1=1 + b∗0‖w‖q1H⇒ b∗0 =‖w‖q1 − ‖w‖p1‖w‖q1 + ‖w‖p1≥ 0. (5)Hence, the ℓq norm is useful in the search for an adequate bias. Note that if p = q, then b∗0 = 0 and the standard bias b isobtained. For this reason, the b∗ is not taken into account in the developments of the standard SVM (p = q = 2) since it isnot necessary. Henceforth, let us call this approach ℓp-SVM-ℓq.With respect to the generalization error, it is known that this barely depends on p when the usual bias is chosen[3,4]. However, the bias in the ℓp-SVM-ℓq, b − b∗0 , is not the standard bias b, and the hyperplane πb∗0 : w′ x + b − b∗0 = 0is geometrically nearer the positive class than the negative class, that is, the bias reduces skew towards the positive class.Hence, the performance on the negative class is increased but it is reduced on the positive class. For this reason, an empiricalstudy is going to be carried out in Section 5.3. Particular cases of ℓp-SVM-ℓqIn this section, some known approaches as particular cases of the ℓp-SVM-ℓq approach are obtained.3.1. ℓ1-SVM-ℓqIn this case, for any q ≥ 1 the optimization problem is as follows:minw,bmaxi=1,...,d|wi|s. t. yiw′ xi + b≥ 1, ∀i = 1, . . . , n.Furthermore,max1≤p≤qmaxw,b,b∗1 − b∗‖w‖p1+1 + b∗‖w‖q1=2‖w‖∞=2maxi=1,...,d|wi|.That is, given Z+ and Z− linearly separable sets, the maximal margin for any p, q ≥ 1 is obtained for p = 1.3.2. ℓ∞-SVM-ℓqIf p = ∞, then the unique possibility of q is q = +∞ and, therefore, the metric is the same in both regions. Theoptimization problem is as follows:minw,bd−i=1|wi|s. t. yiw′ xi + b≥ 1, ∀i = 1, . . . , n(6)which is one of the most widely used approaches of SVM since the optimization problem is linear.3.3. ℓ2-SVM-ℓqIn this case, q ≥ 2 and the optimization problem is as follows:minw,b12‖w‖22s. t. yiw′ xi + b≥ 1, ∀i = 1, . . . , n(7)which is a quadratic optimization problem.It is worth bearing in mind that the most common approach, called the standard primal SVM 2-norm formulation [12],is obtained for q = 2, and the binary linear classifier becomes f(w,b)(x) = π0 = w′ x+ b, that is b∗ = 0. Hence, the selectionof a different bias in [13,14] can be justified from the point of view of an ℓq norm with q ≥ 2 which is explored in Section 5.Usually, the most commonly used norms are ℓ2- or ℓ∞ norms and the same norm is considered in the positive andnegative examples. This fact is due to the algorithms which efficiently solve these optimization problems. Hence, we thinkthat the optimization problem (7) is more relevant than the optimization problem (6) because (i) it is only possible toconsider b∗ = 0 in the classification problem with ℓ∞ norm, and (ii) it is possible to choose different values of ℓq norm(q ≥ 2) in problem (7) which justifies the selection of the b∗.L. Gonzalez-Abril et al. / Computers and Mathematics with Applications 61 (2011) 2874–2878 2877Table 1Results of the experiment where the best mean accuracy rates are presented.p+ q− Glass Iris Tiroide Ecoli2 ∞ 50.282 93.867 95.869 85.2122 10 50.235 93.867 95.869 85.2122 5 49.953 93.933 95.869 85.2122 2.5 51.409 94.533 95.634 85.3642 2.1 51.925 94.400 95.681 85.5152 2 58.263 94.400 95.681 85.5762.1 2 58.545 94.400 95.728 85.6672.5 2 58.498 94.267 95.634 85.8185 2 58.404 93.933 95.446 85.78810 2 58.216 93.867 95.399 85.788∞ 2 58.216 93.867 95.399 85.7884. The extension of ℓp-SVM-ℓqIn the same way as in the standard SVM, the ℓp-SVM-ℓq can be generalized to solve nonlinearly separable andmulticlassification problems. Let us turn our attention to these topics.4.1. Nonlinearly separable caseIf some errors are allowed for the constraints [15], then the optimization problem of ℓp-SVM-ℓq can bewritten as follows:minw,b12‖w‖p1p1 + C−iξis. t. yiw′ xi + b+ ξi ≥ 1, ξi ≥ 0, ∀i = 1, . . . , nwhere C is the regularization term and ξi are slack variables.Note that in the ℓp-SVM-ℓq approach, an upper bound of the number of errors in the classification problem is 2∑ξi.Nevertheless, if p = q, then in the same way as in the standard approach,∑ξi becomes an upper bound.4.2. MulticlassificationA set of possible labels {θ1, . . . , θℓ} (i.e. an unordered set of classes), with ℓ > 2, is considered. Let Z be a training set.Subsets Zk ∈ Z, defined as Zk = {zi = (xi, yi) : yi = θk}, generate a partition in Z. Let us suppose that the norm given in Zkis ℓpk norm with pk ≥ 1 for any k.The 1-v-r SVM and the 1-v-1 SVM approaches [16] are two of themost commonly used alternatives inmulticlassificationproblems and can be used jointly with the ℓp-SVM-ℓq. In both approaches, each machine solves bi-classification problemsand the labels distribution generated by the trained machines in the parallel decomposition is into consideration, througha merging scheme which does not depend on the norm used.5. ExperimentIn this section, the comparison between different bias obtained from (5) is conducted on four widely used data sets fromUCI Repository.4 The experiment has been carried out by following a similar experimental framework to that used in [13].The selected data sets are: Glass Identification Database, Iris Plants, Thyroid Disease and Protein Localization Sites (Ecoli).Performance for the 1-v-r SVM, in the form of accuracy rate, has been evaluated onmodels using the linear kernel whichhas been chosen as a baseline for the empirical evaluation, and C is explored on a one-dimensional grid with the followingvalues: C =2−2, 2−1, . . . , 29, 210. The criteria employed to estimate the generalized accuracy is the threefold cross-validation on thewhole set of training data. This procedure is repeated 20 times in order to ensure good statistical behaviour.It is worth bearing in mind that the generalization error barely depends on p [3,4], the standard primal SVM 2-normformulation is always used in this experimentation. Thus, the cross-validation mean rate for the values of C is reported inTable 1 for the ℓp-SVM-ℓq approach with different values of p and q.Some analysis can be completed according to the empirical experimentation carried out: (i) The accuracy rate attainedfor the standard approach (p = q = 2) can be improved by using a different bias. (ii) No single bias stands out as being thebest. (iii) It has been observed that the difference in the performance between close parameter C is small. (iv) It can be seenthat the generalization error barely depends on the bias.4 Available at http://www.ics.uci.edu/~mlearn/MLRepository.html.2878 L. Gonzalez-Abril et al. / Computers and Mathematics with Applications 61 (2011) 2874–28786. ConclusionsThe SVM 2-norm formulation in the same way as in other approaches based on SVM can be seen as a particular case ofthe approach development in this paper. The classification problem (4) in regions with different norms does not depend onthe ℓq norm if q ≥ p and regardless of whether the ℓp norm is considered in positive or negative examples.Furthermore, the selection of a different bias yields specific results related with the different metrics in each region.Clearly, the selection of p depends on the optimization problem and for simplicity this problem is usually solved by usingp = 2 or p = ∞.On the other hand, the kernel ‘trick‘ can improve this approach in those cases where the solution can be written as alinear combination of the training vector, as happens for p = 2 and p = ∞.References[1] V. Vapnik, Statistical Learning Theory, John Wiley & Sons, Inc, 1998.[2] N. Cristianini, J. Shawe-Taylor, An Introduction to Support Vector Machines and Other Kernel-based Learning Methods, Cambridge University press,2000, 2000.[3] J.P. Pedroso, N. Murata, Support vector machines with different norms: motivation, formulations and results, Pattern Recognition Letters 22 (2001)1263–1272.[4] K. Ikeda, N. Murata, Geometrical properties of Nu Support Vector Machines with different norms, Neural Computation 17 (11) (2005) 2508–2529.[5] F. Love, J. Walker, An empirical comparison of block and round norms for modelling actual distances, Location Science 2 (1994) 21–43.[6] M. Parlar, Single facility location problem with region-dpendent distance metrics, International Journal of Systems Sciences 25 (3) (1994) 513–525.[7] J. Brimberg, Locating a single facility in the plane in the presence of a bounded region and different norms, Annals of Operations Research 122 (2003)87–102.[8] J. Brimberg, H. Taghizadeh, Kakhki, G. Wesolowsky, Locating a single facility in the plane in the presence of a bounded region and different norms,Journal of Operational Research Society of Japan 48 (2) (2005) 135–147.[9] M. Zaferanieh, H.T. Kakhki, J. Brimberg, G. Wesolowsky, A bsss algorithm for the single facility location problem in two regions with different norms,European Journal of Operational Research 190 (1) (2008) 79–89.[10] O.L. Mangasarian, Arbitrary-norm separating plane, Operations Research Letters 24 (1999) 15–23.[11] A. Dax, The distance between two convex sets, Linear Algebra and its Applications 416 (2006) 184–213.[12] L. González, C. Angulo, F. Velasco, A. Català, Unified dual for bi-class SVM approaches, Pattern Recognition 38 (10) (2005) 1772–1774.[13] L. Gonzalez-Abril, C. Angulo, F. Velasco, J. Ortega, A note on the bias in SVMs for multi-classification, IEEE Transactions on Neural Networks 19 (4)(2008) 723–725.[14] H. Nuñez, L. Gonzalez-Abril, C. Angulo, A post-processing strategy for SVM learning from unbalanced data, in: Proceedings of the 19th EuropeanSymposium on Artificial Neural Networks, ESANN, 2011.[15] L. González, C. Angulo, F. Velasco, A. Català, Dual unification of bi-class support vector machine formulations, Pattern Recognition 39 (7) (2006)1325–1332.[16] C. Angulo, F. Ruiz, L. González, J.A. Ortega, Multi-classification by using tri-class SVM, Neural Proceeding Letters 23 (1) (2006) 89–101.

Support vector machines for classification of input vectors with different metrics

https://idus.us.es/bitstream/11441/142972/1/Support%20vector%20machines%20for%20classification%20of%20input%20vectors%20with%20different%20metrics.pdf

Support vector machines for classification of input vectors with different metrics

Abstract

Similar works

Full text

Available Versions

idUS. Depósito de Investigación Universidad de Sevilla