The k-support norm has been recently introduced to perform correlated
sparsity regularization. Although Argyriou et al. only reported experiments
using squared loss, here we apply it to several other commonly used settings
resulting in novel machine learning algorithms with interesting and familiar
limit cases. Source code for the algorithms described here is available

Blaschko, Matthew

English

arXiv

The k-support norm has been recently introduced to perform correlated sparsity regularization. Although Argyriou et al. only reported experiments using squared loss, here we apply it to several other commonly used settings resulting in novel machine learning algorithms with interesting and familiar limit cases. Source code for the algorithms described here is available

HAL-CentraleSupelec

A Note on k-support Norm Regularized Risk Minimization

INRIA a CCSD electronic archive server

HAL Id: hal-00804592https://hal.inria.fr/hal-00804592v2Preprint submitted on 27 Mar 2013HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.A Note on k-support Norm Regularized RiskMinimizationMatthew BlaschkoTo cite this version:Matthew Blaschko. A Note on k-support Norm Regularized Risk Minimization. 2013. ￿hal-00804592v2￿A Note on k-support Norm Regularized RiskMinimizationMatthew B. BlaschkoÉcole Centrale ParisGrande Voie des Vignes92295 Châtenay-Malabry, Francematthew.blaschko@inria.frAbstract. The k-support norm has been recently introduced to performcorrelated sparsity regularization [1]. Although Argyriou et al. only re-ported experiments using squared loss, here we apply it to several othercommonly used settings resulting in novel machine learning algorithmswith interesting and familiar limit cases. Source code for the algorithmsdescribed here is available from https://github.com/blaschko/ksupport.Keywords: k-support norm, structured sparsity, regularization, least-squares,hinge loss, support vector machine, SVM, regularized logistic regression, Ad-aBoost, support vector regression, SVR1 The k-support NormThe k-support norm is the gauge function associated with the convex setconv{β | ‖β‖0 ≤ k, ‖β‖2 ≤ 1}. (1)It can be computed as‖β‖spk =k−r−1∑i=1(|β|↓i )2 +1r + 1(d∑i=k−r|β|↓i)2 12 (2)where |β|↓i is the ith largest element of the vector and r is the unique integer in{0, . . . , k − 1} satisfying|β|↓k−r−1 >1r + 1d∑i=k−r|β|↓i ≥ |β|↓k−r. (3)We use the following notation here: X ∈ Rn×d is a design matrix of n sampleseach with d dimensions; y ∈ Rn is the vector of targets.In the case that k = 1 the k-support norm is exactly equivalent to the `1norm. In the case that k = d, where β ∈ Rd, the k-support norm is equivalentto the `2 norm.We note that for an objectiveminβλ‖β‖spk + f(β,X, y) (4)with some loss function f(·, ·, ·), when k = d, this is equivalent tominβλ‖β‖2 + f(β,X, y) (5)rather than the familiar squared `2 regularizer. However, for any λ there existssome λ̃ such thatarg minβλ‖β‖2 + f(β,X, y) = arg minβλ̃‖β‖22 + f(β,X, y). (6)This can be easily seen by noting that the objectives are the Lagrangians ofconstrained minimization problems that minimize f subject to the equivalentconstraints ‖β‖2 ≤ B and ‖β‖22 ≤ B2, respectively, for some B ∈ R+.2 Squared LossIf we use Nesterov’s accelerated method (a first-order proximal algorithm) foroptimization as suggested in [1], a given implementation of k-support regularizedrisk requires a function that computes the loss f , a function that computes thegradient of the loss function ∂f∂β , and the Lipschitz constant L for∂f∂β . We assumethat f is convex and differentiable everywhere and that L is finite.For the squared loss:f2(β,X, y) = ‖Xβ − y‖2 (7)∂f2∂β= 2XTXβ − 2XT y (8)L2 = 2γ (9)where γ is the largest eigenvalue of XTX.The objective functionλ‖β‖spk + ‖Xβ − y‖2 (10)clearly has the lasso [11] and ridge regression [12] as special cases when k = 1 andk = d, respectively. Argyriou et al. [1] have previously discussed the relationshipto the elastic net [17]. The k-support norm with squared loss has been shown togive good results on fMRI data [7].3 One Sided Squared LossWhile we have previously assumed that y ∈ Rn, here we will assume we aredealing with the binary classification case where y ∈ {−1,+1}n. One sidedsquared loss simply computes the squared loss when a margin is violated, andzero otherwise.f2−(β,X, y) =n∑i=1(max{1− yi〈β, xi〉, 0})2 (11)∂f2−∂β=n∑i=1{0 if yi〈β, xi〉 > 12〈β, xi〉xi − 2yixi if yi〈β, xi〉 ≤ 1(12)L2− = 2γ. (13)One sided squared loss has been considered, for example, in [4].4 Hinge LossHinge loss is not differentiable, so we apply a Huber approximation to hingeloss [4].1 The Huber parameter is denoted h:fh(β,X, y) =n∑i=10 if yi〈β, xi〉 > 1 + h(1+h−yi〈β,xi〉)24h if |1− yi〈β, xi〉| ≤ h1− yi〈β, xi〉 if yi〈β, xi〉 < 1− h(14)∂fh∂β=n∑i=10 if yi〈β, xi〉 > 1 + h〈β,xi〉xi−(1+h)yixi2h if |1− yi〈β, xi〉| ≤ h−yixi if yi〈β, xi〉 < 1− h(15)L2 =γ2h(16)where γ is as before the largest eigenvalue of XTX. We note that the Lipschitzconstant is in a sense conservative in that it grows with the inverse of h, while wemight expect a smaller fraction of the data to actually fall within the quadraticportion of the data. Nevertheless for h not too small, we have not observed anyconvergence issues with Nesterov’s accelerated method. While a small value ofh may be desirable in a kernelized setting, here we desire Hinge loss not forsparsity of a dual coefficient vector (indeed the k-support norm does not admita representer theorem [2]), but rather that the loss not grow more than linearlywhile remaining convex. In other words, we use the hinge loss primarily for itsincreased robustness over other losses such as (one-sided) squared loss.The limit cases are the support vector machine (SVM) [5] when k = d andthe `1 regularized SVM [16] when k = 1. The k-support regularized SVM can beseen as an alternative to the elastic net regularized SVM [14], but with a tighterconvex relaxation to correlated sparsity (Equation (1)).1 Although it is perhaps more natural to incorporate non-differentiable losses with thek-support regularizer in a proximal splitting approach, we have arbitrarily closely ap-proximated non-differentiable losses by differentiable ones for the sake of uniformityof presentation and software implementation.5 Logistic LossLogistic loss is derived from logistic regression, and its minimization is equivalentto logistic regression in the case that it is unregularized [8].flog(β,X, y) =n∑i=1log(1 + e−yi〈β,xi〉)(17)∂flog∂β= −n∑i=1e−yi〈β,xi〉1 + e−yi〈β,xi〉yixi (18)Llog =γ4(19)where the Lipschitz constant has a factor 14 from the Lipschitz constant of thesigmoid in∂flog∂β . k-support regularized regression specializes to previously usedregularized logistic regression objectives [9] when k = 1 or k = d.6 Exponential LossExponential loss is known primarily through its use in AdaBoost.M1 [6, 8].fexp(β,X, y) =n∑i=1e−yi〈β,xi〉 (20)∂fexp∂β= −n∑i=1e−yi〈β,xi〉yixi (21)Here, the loss is not globally Lipschitz continuous. However, one may attemptto estimate a sufficiently large constant if one were to apply learning with thek-support norm and Nesterov’s accelerated method (we have simply used a rel-atively conservative 50× γ in the experiments reported in Section 8). As expo-nential loss is highly degenerate in the presence of label noise (essentially for thesame reason that it is not globally Lipschitz continuous), this is likely of limitedutility in real-world applications. We have included this loss here primarily forcompleteness, and have not explored any other optimization strategies.7 ε-insensitive Loss and Huber Smoothed Absolute Lossε-insensitive loss is defined to be [13]:|yi − 〈β, xi〉|ε := max{0, |y − 〈β, xi〉| − ε} (22)for some parameter ε ≥ 0. While ε has an important role in the sparsity ofthe dual representation for support vector regression [13], that role is not re-quired in the primal. As with hinge loss, we use Huber smoothing to guarantee−5 −4 −3 −2 −1 0 1 2 3 4 5−1−0.500.511.522.53(a) ε = 2 gives an insensitive region around the cor-rect regression value.−5 −4 −3 −2 −1 0 1 2 3 4 5−1012345(b) The special case that ε = 0 results in a Hubersmoothed absolute loss.Fig. 1. Huber smoothed ε-insensitive loss. On the horizontal axis is yi − 〈β, xi〉 whilethe vertical axis plots fε in blue, and∂fε∂βin red. In both plots h = 12.Table 1. Accuracies for each method and regularizer. See text for the experimentalsetting. The k-support norm achieved higher acuracies on average for all loss functions.f2 f2− fh flog fexp fabs fε‖β‖spk 0.883± 0.058 0.883± 0.058 0.890± 0.057 0.889± 0.056 0.888± 0.060 0.889± 0.065 0.886± 0.062‖β‖1 0.870± 0.062 0.870± 0.062 0.868± 0.069 0.872± 0.063 0.876± 0.065 0.870± 0.077 0.879± 0.059‖β‖2 0.871± 0.071 0.871± 0.071 0.872± 0.065 0.872± 0.066 0.870± 0.067 0.867± 0.071 0.872± 0.063differentiability.fε(β,X, y) =n∑i=10 if yi − 〈β, xi〉 > −ε+ h(yi−〈β,xi〉+ε−h)24h if |yi − 〈β, xi〉+ ε| ≤ h−yi + 〈β, xi〉 − ε if yi − 〈β, xi〉 < −ε− h(23)+n∑i=10 if yi − 〈β, xi〉 < ε− h(yi−〈β,xi〉−ε+h)24h if |yi − 〈β, xi〉 − ε| ≤ hyi − 〈β, xi〉 − ε if yi − 〈β, xi〉 > ε+ h∂fε∂β=n∑i=10 if yi − 〈β, xi〉 > −ε+ h〈β,xi〉xi+(−ε+h−yi)xi2h if |yi − 〈β, xi〉+ ε| ≤ hxi if yi − 〈β, xi〉 < −ε− h(24)+n∑i=10 if yi − 〈β, xi〉 < ε− h〈β,xi〉xi+(ε−h−yi)xi2h if |yi − 〈β, xi〉 − ε| ≤ h−xi if yi − 〈β, xi〉 > ε+ hLε =γh(25)Here we have decomposed the ε insensitive loss into two hinge components toemphasize the relationship to Huber smoothed hinge loss (cf. Section 4). A plotof the loss and its gradient is shown in Figure 1. In the case that ε = 0 we geta Huber smoothed absolute loss function as a special case (denoted fabs in thesequel), and the curvature of the loss function at yi − 〈β, xi〉 = 0 is doubled,therefore the Lipschitz constant is double that of the one sided hinge loss.In the case that k = d, we recover the special case of ε-support vector regres-sion (ε-SVR) [10]. If we set k = 1 we get an `1 regularized variant of ε-SVR. Inthe case that ε = 0 this `1 regularized variant is equivalent to regularized leastabsolute deviations regression [15]. In Equation (23), ε < 0 is equivalent to ε > 0but with a constant value added to the loss everywhere, i.e. the minimizer is thesame.8 ExperimentsWe have applied each of the algorithms above to a toy classification problem con-ceptually similar to that reported in [1]. In all cases, we perform model selectionfor k ∈ {1, . . . , d} and λ = 10i, i ∈ {−15, . . . , 5}. We compare additionally toTable 2. Mean squared errors (MSE) for each method and regularizer. See text for theexperimental setting. f2, f2−, fabs, and fε achieved the lowest MSEs with the k-supportnorm regularizer giving best results on average.f2 f2− fh flog fexp fabs fε‖β‖spk 1.21e2± 4.89e1 1.21e2± 4.89e1 1.78e2± 1.00e2 3.33e3± 5.39e3 1.59e3± 2.89e3 1.25e2± 5.41e1 2.21e2± 1.51e1‖β‖1 1.25e2± 4.81e1 1.25e2± 4.81e1 2.21e2± 9.63e1 1.13e4± 9.89e3 6.16e3± 4.82e3 1.48e2± 1.76e2 2.16e2± 1.66e1‖β‖2 1.49e2± 4.75e1 1.49e2± 4.74e1 1.81e2± 7.66e1 4.18e3± 8.00e3 3.08e3± 4.88e3 1.50e2± 5.34e1 2.25e2± 1.56e1the special fixed cases k = 1 and k = d corresponding to `1 and `2 regularization,respectively.Output labels were generated randomly with equal probability. The first 15dimensions were set by multiplying the label by a fixed vector of 15 samples froma zero mean Gaussian and adding Gaussian noise (i.e. a noisy signal is containedin the first 15 dimensions). The subsequent 50 dimensions were set to zero meanGaussian noise (i.e. the subsequent dimensions contain no signal and should beignored). 50 samples were used for training, 50 for validation, and 250 for testing.Table 1 gives the mean accuracies for each method across 20 random probleminstances, while Table 2 gives the mean squared error (MSE). For ε-insensitiveloss we arbitrarily set ε = 1. For all methods with a Huber smothing parameter,we set h = 110 .It should be noted that several of the methods employed here for classificationwere developed for regression (squared loss, absolute loss, and ε-insensitive loss).The experiments performed here were done primarily to validate their correctimplementation.9 ConclusionsWe have described and implemented a large number of loss functions for non-differentiably regularized risk optimization with proximal splitting methods.These loss functions in combination with the k-support norm yield a large num-ber of learning algorithms proposed in the literature as special cases. Assumingzero model error, each of these loss functions is sufficient to yield a statisticallyconsistent algorithm2 (provided regularization goes to zero at a sufficient rateas the number of samples goes to infinity) [3, Theorem 4]. However, their finitesample behavior varies substantially. We hope that their implementation and de-scription in a common framework will facilitate their analysis and employmentin machine learning studies and applications.References1. Argyriou, A., Foygel, R., Srebro, N.: Sparse prediction with the k-support norm.In: NIPS. pp. 1466–1474 (2012)2 Huber smoothed ε-insensitive loss requires that ε − h < 1 for consistency in thebinary classification setting, yi ∈ {−1,+1}.2. Argyriou, A., Micchelli, C.A., Pontil, M.: When is there a representer theorem?vector versus matrix regularizers. Journal of Machine Learning Research 10, 2507–2529 (2009)3. Bartlett, P.L., Jordan, M.I., McAuliffe, J.D.: Convexity, classification, and riskbounds. Journal of the American Statistical Association 101(473), 138–156 (2006)4. Chapelle, O.: Training a support vector machine in the primal. Neural Computa-tion 19(5), 1155–1178 (May 2007)5. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (Sep1995)6. Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: ICML.pp. 148–156 (1996)7. Gkirtzou, K., Honorio, J., Samaras, D., Goldstein, R., Blaschko, M.B.: fMRI anal-ysis of cocaine addiction using k-support sparsity. In: ISBI (2013)8. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning, Sec-ond Edition: Data Mining, Inference, and Prediction. Springer Series in Statistics,Springer (2009)9. Ng, A.Y.: Feature selection, l1 vs. l2 regularization, and rotational invariance. In:Proceedings of the twenty-first international conference on Machine learning (2004)10. Schölkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Reg-ularization, Optimization, and Beyond. MIT Press, Cambridge, MA, USA (2001)11. Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of theRoyal Statistical Society (Series B) 58, 267–288 (1996)12. Tikhonov, A.: Solution of incorrectly formulated problems and the regularizationmethod. In: Soviet Math. Doklady. vol. 4, pp. 1035–1038 (1963)13. Vapnik, V.N.: The nature of statistical learning theory. Springer-Verlag New York,Inc., New York, NY, USA (1995)14. Wang, L., Zhu, J., Zou, H.: The doubly regularized support vector machine. Sta-tistica Sinica 16(2), 589–616 (2006)15. Wang, L., Gordon, M.D., Zhu, J.: Regularized least absolute deviations regres-sion and an efficient algorithm for parameter tuning. In: Proceedings of the SixthInternational Conference on Data Mining. pp. 690–700 (2006)16. Zhu, J., Rosset, S., Hastie, T., Tibshirani, R.: 1-norm support vector machines. In:Thrun, S., Saul, L., Schölkopf, B. (eds.) Advances in Neural Information ProcessingSystems 16. MIT Press, Cambridge, MA (2004)17. Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. Jour-nal of the Royal Statistical Society, Series B 67, 301–320 (2005)

A Note on k-support Norm Regularized Risk Minimization

Abstract

Similar works

Full text

Available Versions

HAL-CentraleSupelec

INRIA a CCSD electronic archive server