Fabio, Aiolli

Ivano, Lauriola

Polato, Mirko

English

Institutional Research Information System University of Turin

The Minimum Effort Maximum OutputPrinciple applied to Multiple Kernel LearningIvano Lauriola∗, Mirko Polato and Fabio AiolliUniversity of Padova - Department of MathematicsVia Trieste 63, Padova - ItalyAbstract. The Multiple Kernel Learning (MKL) paradigm aims atlearning the representation from data reducing the effort devoted to thechoice of the kernel’s hyperparameters. Typically, the resulting kernelis obtained as the maximal margin combination of a set of base kernels.When too expressive base kernels are provided to the MKL algorithm,the solution found by these algorithms can overfit data. In this paper,we propose a novel MKL algorithm which takes into consideration theexpressiveness of the obtained representation in its objective function insuch a way that a trade-off between large margins and simple hypothesisspaces can be found. Moreover, an empirical comparison against hardbaselines and state-of-the-art MKL methods on several real-world datasetsis presented showing the merits of the proposed algorithm especially withrespect to the robustness to overfitting.1 IntroductionKernel methods are recognized state-of-the-art methods for classification. Theyrely on the concept of kernel which implicitly defines the data representation.However, choosing the right kernel for a given problem remains an hard task. In atypical learning pipeline the user tries several kernels with different values of thehyperparameters, guided by some prior knowledge or via a validation procedure.This process is computationally expensive when the number of possible valuesfor the kernels hyperparameters is large. To overcome this issue, methods todirectly learn kernels from data have been recently proposed. Multiple KernelLearning (MKL) [1] is one of the most popular frameworks for kernel learning[2, 3, 4, 5, 6]. The idea behind these methods is to combine a set of base kernels.This is usually done by finding the kernel combination that maximizes the marginin feature space. However, the complexity (or expressiveness) of the inducedrepresentation is generally neglected, with the risk of obtaining a too complexrepresentation for the task at hand. To overcome this problem, more recentapproaches tried to regularize the combination, for instance, by considering theradius of the Minimum Enclosing Ball (MEB) in their objective function [7, 8].The main contribution of this paper is the proposal of a novel MKL algorithm,here dubbed MEMO-MKL (Minimum Effort Maximum Output MKL), whichfinds a combination of base kernels trading-off between margin maximization andlow expressiveness of the resulting representation. An extensive experimentalassessment has been performed comparing the proposed method against strongbaselines/state-of-the-art MKL methods on several benchmark datasets.∗The PhD grant of Ivano Lauriola is funded by the Bruno Kessler Foundation.ESANN 2018 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 25-27 April 2018, i6doc.com publ., ISBN 978-287587047-6. Available from http://www.i6doc.com/en/.  1832 Background and notationWe focus on binary classification problems. Let D ≡ {(xi, yi)}li=1 be the trainingset where xi ∈ Rn are n-dimensional input vectors and yi ∈ {+1,−1} are theassociated labels. A kernel function κ : X × X → R defines the dot-productin a Reproducing Kernel Hilbert Space (RKHS) H, by means of the functionφ : X → H, which maps vectors from the input space X onto the embedding (orfeature) space H, such that κ(x, z) = 〈φ(x), φ(z)〉 for two generic input vectorsx and z. The Kernel (or Gram) matrix K ∈ Rl×l is the matrix containingthe evaluation of the kernel function for all pairs of training vectors. Let Kbe a kernel matrix, the (squared) margin achieved by a hard-SVM in featurespace can be computed as the value of the objective of a quadratic optimizationproblem, namely minγ∈Γ γᵀYKYγ, where Y is the diagonal matrix containingthe labels and Γ = {γ ∈ Rl+|∑i:yi=+1γi =∑i:yi=−1 γi = 1}. In the following,the Homogeneous Polynomial Kernel (HPK) κdHP(x, z) = 〈x, z〉d, d ∈ N and itsnormalized version κ̃dHP(x, z) =κdHP(x,z)√κdHP(x,x)κdHP(z,z)will be used as base kernels.2.1 Multiple Kernel LearningMKL [1] is a method to combine a set of base kernels that represent differentsimilarity measures or different data sources (e.g. audio and video). Manyfunctional forms exist to combine kernels. In this paper, the convex combinationof base kernels is considered, that is:κµ(x, z) =P∑r=1µrκr(x, z) =P∑r=1µr〈φr(x), φr(z)〉, µr ≥ 0,∑rµr = 1.where the r-th kernel κr(x, z) = 〈φr(x), φr(z)〉 consists of the dot-product infeature space defined by the mapping function φr. In particular, we considerHPKs of increasing degrees as base kernels. It has been shown that, under mildconditions, any dot-product kernel of the form f(〈x, z〉) can actually be seen asa convex combination of HPKs [9]. In other words, applying MKL on HPKs canbe seen as selecting a representation from the space of dot-product kernels.2.2 Expressiveness of base kernelsThe expressiveness (or complexity) of a kernel function can be defined as thenumber of dichotomies that can be realized by hyperplanes inside such featurespace. It has been demonstrated that the expressiveness of a kernel is relatedto the rank of its induced kernel matrix. Specifically, given a set of instances Sand the associated kernel matrix of rank r, then there is at least r instances inS that can be fragmented in the feature space [9]. However, using the rank ofthe kernel matrix for estimating the expressiveness can be very expensive sinceit requires the decomposition of the kernel matrix. To this end, in [9], a simplermeasure able to approximate the rank of a kernel has been proposed, namely theESANN 2018 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 25-27 April 2018, i6doc.com publ., ISBN 978-287587047-6. Available from http://www.i6doc.com/en/.  184Spectral Ratio (SR). Specifically, given a kernel matrix K, the SR is the ratiobetween the trace norm ‖ · ‖T and the Frobenius norm ‖ · ‖F of K:C(K) = ‖K‖T‖K‖F=∑iKii√∑ijK2ij. (1)In the same work, it has been shown that 1 ≤ C(K) ≤√rank(K) alwaysholds. Furthermore, the SR has been related to other complexity measures,such as the Empirical Rademacher Complexity and the radius of the MinimumEnclosing Ball (MEB) in the feature space, that is the radius of the smallesthypersphere enclosing the data in the kernel space. Note that, the maximal SRis obtained by the identity kernel matrix, where examples are orthogonal to eachother C(Il) =√l, whereas the constant kernel C(1l1ᵀl ) = 1 has the minimal SR.Notably, when a kernel is normalized (‖φ(x)‖2 = 1) the SR formulation in (1)can be further simplified, that is C(K) = l‖K‖−1F .3 MEMO-MKLMKL algorithms generally aim at finding mixing coefficients maximizing themargin between the positive and negative classes. One of the main issues of MKLis that the combined kernel may be too expressive for a given problem, with therisk of overfitting. To overcome this problem a regularization term is usuallyincluded in the optimization criterion of these algorithms. However, as far as weknow, only the mixing coefficients are regularized while no terms related to theresulting kernel complexity is taken into consideration in the objective function.In the MEMO-MKL algorithm presented in this paper we propose to find themixing coefficients µ which jointly minimize the SR and maximize the margin.Note that, normalized kernels and convex combinations of them have constanttrace norm. Hence, minimizing the SR corresponds to maximizing the Frobeniusnorm, that can be characterized for a combined kernel as in the following:‖Kµ‖2F =∑i,j(∑rµrK(r)ij)2=∑i,j∑r,sµrµsK(r)ij K(s)ij=∑r,sµrµs∑i,jK(r)ij K(s)ij︸ ︷︷ ︸Qrs= µᵀQµ, Q ∈ RP×P .In order to make the MKL problem unconstrained, a new vector of variables βis introduced, such that µr(β) = eβr/||eβ||1 and Kµ(β) =∑Pr=1 µr(β)K(r).Given the change of variables above, the proposed algorithm maximizes thefollowing unconstrained objective function, that is a trade-off between simplerrepresentation (low SR) and large margin solutions:Ψ(β) = γ̂(β)ᵀYKµ(β)Yγ̂(β) +θ2µ(β)ᵀQµ(β), (2)ESANN 2018 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 25-27 April 2018, i6doc.com publ., ISBN 978-287587047-6. Available from http://www.i6doc.com/en/.  185where γ̂(β) = arg minγ∈ΓγᵀYKµ(β)Yγ, (3)Note that, when θ = 0, the complexity of the representation becomes irrele-vant, and the algorithm will find the coefficients maximizing the margin.In order to maximize Ψ(β) which is not convex, an iterative optimization algo-rithm is applied. We start from the uniform distribution of µ(β), correspondingto β = 0. At each step, the algorithm first finds the coefficients γ̂(β) by op-timizing (3) with the current β. Then, a step of gradient ascent is performedon Ψ(β) where the vector γ̂ = γ̂(β) is considered locally constant. With theseassumptions, the derivatives of Ψ(β) can be easily computed as∂Ψ(β)∂βr= γ̂ᵀYK(r)Yγ̂ + θµ(β)ᵀQr.and the mixing weights updated by βr ← βr +η ∂Ψ(β)∂βr , µr(β) = eβr/||eβ||1, ∀r ∈{1, . . . , P}. The learning rate η is selected dinamically during the optimizationby using the backtracking line-search method. The optimization loop ends whena maximum number of iterations is reached or the improvement of the objectivefunction is below a fixed threshold.4 Experimental assessmentTo evaluate the effectiveness of the proposed MKL algorithm, a comparisonagainst hard baselines has been performed on several benchmark problems, de-scribed in Table 1. These datasets are freely available on the UCI1 and libsvmrepositories2.The datasets were randomly split in training (70%) and test (30%) sets, andfeatures were rescaled between 0 and 1 to prevent negative values. Multiclassproblems have been mapped into binary ones keeping the distribution of positiveand negative examples balanced. A 5-fold cross validation procedure on thetraining set has been used to find the best combination of the hyperparameters.After fitting the models, the accuracy score is computed on the test set. Thisprocedure has been applied 30 times, and the average accuracies were recorded.Finally, normalized HPKs with degrees d ∈ [1, 20], and the identity matrix kernelare considered for combination by MKL. The compared MKL methods are:• average of base kernels, which is known to be a hard baseline, µr = 1P ;• EasyMKL, a fast and effective margin maximization MKL algorithm [10];• (fixed) linear kernel, µ1 = 1, µi = 0 ∀i 6= 1;• MEMO-MKL, the algorithm proposed in this paper.1M. Lichman. UCI machine learning repository, 2013.2https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/ESANN 2018 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 25-27 April 2018, i6doc.com publ., ISBN 978-287587047-6. Available from http://www.i6doc.com/en/.  186In all the experiments, an hard-margin SVM is used as base learner. Thehyperparameters are λ ∈ {0, .01, .1, .2, .3, .9, .95, 1} for EasyMKL, and θ ∈ {10i :−4 ≤ i ≤ 4} for MEMO-MKL. Results are summarized in Table 1, where theaverage of accuracies, standard deviations, datasets informations, and the ranksof all used algorithms are shown. The results clearly show that MEMO-MKLis better than EasyMKL when the base kernels are too complex for the taskconsidered and this especially happens when the number of features is largewith respect to the number of examples.dataset l × n average EasyMKL MEMO-MKL linearduke 44× 7129 59.70±19.87 64.24±19.77 84.55±9.72 84.55±9.72colon 62× 2000 65.83±9.78 71.67±13.38 83.96±7.85 83.75±7.50leukemia 72× 7129 64.81±8.53 76.11±11.84 96.85±3.71 96.85±3.71liver-disorder 145× 5 65.59±5.75 68.29±5.75 70.00±7.46 73.87±5.51iris 150× 4 89.30±4.75 91.67±5.22 89.47±4.80 89.21±4.47wine 178× 13 97.63±3.39 97.78±3.14 97.78±1.99 97.33±2.10sonar 208× 60 85.58±5.61 85.64±5.63 86.15±5.40 73.59±3.37glass 214× 9 77.84±5.69 77.59±6.17 78.58±5.47 69.20±6.51heart 270× 13 77.25±3.94 78.28±3.68 80.39±4.17 82.25±4.26ionosphere 351× 34 96.36±1.59 96.33±1.49 96.33±1.57 89.05±3.51breast-cancer 683× 9 96.47±1.33 96.41±1.23 96.20±1.24 96.32±1.47australian 690× 14 81.23±2.92 84.89±2.81 85.18±2.73 84.74±2.50diabetes 768× 8 74.25±2.31 76.77±2.56 77.19±2.24 77.27±2.44vehicle 846× 18 96.98±1.17 97.11±1.20 97.11±1.23 91.15±1.61fourclass 862× 2 80.11±2.41 79.95±2.58 80.29±2.64 75.96±3.00vowel 990× 10 99.37±0.58 99.35±0.54 99.34±0.56 85.55±1.82german-number 1000× 24 73.33±2.47 73.09±2.61 76.47±2.57 76.73±2.36splice 1000× 60 84.85±1.54 84.93±2.12 84.80±1.55 80.05±2.27dna 2000× 180 93.97±0.85 94.61±0.84 94.12±0.83 89.91±1.41madelon 2000× 500 57.80±1.82 59.55±1.81 57.78±1.81 54.58±2.15rank 2.85 2.20 1.80 2.90Table 1: Datasets informations, accuracy scores (%) and rank of the proposedmethod and baselines. Best results are marked as bold.The weights computed by MEMO-MKL are analyzed in Fig. 1 (left), showingtheir behavior when the hyperparameter θ is changed. As expected, increasingthe value θ, the distribution of weights is concentrated on low degree HPKs (lowerexpressiveness), whereas more expressive HPKs will have higher weights whenthe algorithm optimizes the margin (lower θs). In Fig. 1 (right) a comparisonbetween the weights computed by MEMO-MKL with θ = 0 and EasyMKL isdepicted, showing different distributions when both the methods focuses on largemargin solutions.5 Conclusions and future workIn this paper a novel MKL algorithm have been proposed which finds the kernelscombination that maximizes the margin keeping low the complexity of the re-sulting representation. The algorithm has been evaluated empirically on severalbenchmark problems, showing better accuracies compared with three baselines.Furthermore, the solution computed is generally very sparse, and the weightsESANN 2018 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 25-27 April 2018, i6doc.com publ., ISBN 978-287587047-6. Available from http://www.i6doc.com/en/.  1870 10 2000.51weak kerneldukeθ:1.0 θ:1.3 θ:1.50 10 2005 · 10−20.10.150.2weak kernelweightdukeEasyMKL MEMO-MKL (θ = 0)Fig. 1: Weights computed by MEMO-MKL varying the value θ (left) and weightscomparison between EasyMKL and MEMO-MKL with θ = 0 (right).depend strictly on the selection of the regularization hyperparameter θ. In thefuture, the behavior of the proposed algorithm when using different kernel func-tions will be better analyzed, a comparison against other MKL approaches willbe considered, and the sparsity property will be more theoretically studied.References[1] Mehmet Gönen and Ethem Alpaydın. Multiple kernel learning algorithms. Journal ofMachine Learning Research, 12(Jul):2211–2268, 2011.[2] Soujanya Poria, Haiyun Peng, Amir Hussain, Newton Howard, and Erik Cambria. En-semble application of convolutional neural networks and multiple kernel learning for mul-timodal sentiment analysis. Neurocomputing, 2017.[3] Nora K Speicher and Nico Pfeifer. Integrating different data types by regularized unsu-pervised multiple kernel learning with application to cancer subtype discovery. Bioinfor-matics, 31(12):i268–i275, 2015.[4] Blake Anderson, Curtis Storlie, and Terran Lane. Multiple kernel learning clusteringwith an application to malware. In Data Mining (ICDM), 2012 IEEE 12th InternationalConference on, pages 804–809. IEEE, 2012.[5] Ivano Lauriola, Michele Donini, and Fabio Aiolli. Learning dot-product polynomials formulticlass problems. In Proceedings of the European Symposium on Artificial NeuralNetworks, Computational Intelligence and Machine Learning (ESANN), 2017.[6] Alexander Zien and Cheng Soon Ong. Multiclass multiple kernel learning. In Proceedingsof the 24th international conference on Machine learning, pages 1191–1198. ACM, 2007.[7] Ivano Lauriola, Mirko Polato, and Fabio Aiolli. Radius-margin ratio optimization fordot-product boolean kernel learning. In International Conference on Artificial NeuralNetworks (ICANN), 2017.[8] Alexandros Kalousis and Huyen T. Do. Convex formulations of radius-margin basedsupport vector machines. In Proceedings of the 30th International Conference on MachineLearning, volume 28, pages 169–177, 2013.[9] Michele Donini and Fabio Aiolli. Learning deep kernels in the space of dot productpolynomials. Machine Learning, pages 1–25, 2016.[10] Fabio Aiolli and Michele Donini. Easymkl: a scalable multiple kernel learning algorithm.Neurocomputing, pages 215–224, 2015.ESANN 2018 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 25-27 April 2018, i6doc.com publ., ISBN 978-287587047-6. Available from http://www.i6doc.com/en/.  188

The minimum effort maximum output principle applied to Multiple Kernel Learning

https://iris.unito.it/bitstream/2318/1870181/1/es2018-181.pdf

The minimum effort maximum output principle applied to Multiple Kernel Learning

Abstract

Similar works

Full text

Available Versions

Institutional Research Information System University of Turin