Biehl, M.

Geweniger, T.

Schleif, F.-M.

Schneider, P.

Villmann, T.

English

ARTS repository - University of Groningen

   University of GroningenMultivariate class labeling in Robust Soft LVQSchneider, P.; Geweniger, T.; Schleif, F.-M.; Biehl, M.; Villmann, T.Published in:19th European Symposium on Artificial Neural Networks (ESANN 2011)IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2011Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Schneider, P., Geweniger, T., Schleif, F-M., Biehl, M., & Villmann, T. (2011). Multivariate class labeling inRobust Soft LVQ. In M. Verleysen (Ed.), 19th European Symposium on Artificial Neural Networks (ESANN2011) (pp. 17-22). d-side publishing.CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 21-06-2022Multivariate class labeling in Robust Soft LVQPetra Schneider1, Tina Geweniger2, Frank-Michael Schleif3,Michael Biehl4 and Thomas Villmann21- School of Clinical and Experimental Medicine - University of BirminghamBirmingham B15 2TT - United Kingdom2- Department of MPI - University of Applied Sciences MittweidaTechnikumplatz 17 - 09648 Mittweida - Germany3- CITEC - University of BielefeldUniversitätsstrasse 21-23 - 33615 Bielefeld - Germany4- Institute for Mathematics and Computer Science - University of GroningenP.O. Box 407 - 9700 AK Groningen - The NetherlandsAbstract. We introduce a generalization of Robust Soft Learning VectorQuantization (RSLVQ). This algorithm for nearest prototype classificationis derived from an explicit cost function and follows the dynamics of astochastic gradient ascent. We generalize the RSLVQ cost function withrespect to vectorial class labels: Probabilistic LVQ (PLVQ) allows to re-alize multivariate class memberships for prototypes and training samples,and the prototype labels can be learned from the data during training. Wepresent experiments to demonstrate the new algorithm in practice.1 IntroductionLearning Vector Quantization (LVQ) is a supervised classification scheme whichwas introduced by Kohonen in 1986 [1]. The approach still enjoys great popu-larity and numerous modifications of the original algorithm have been proposed(see e.g. [2, 3, 4]). The algorithms learn a set of class-specific prototype vec-tors which represent the data in the input space. The set of labeled prototypesparameterizes a nearest-prototype classification, e.g. an unknown pattern is as-signed to the class represented by the closest prototype with respect to somedistance metric. The popularity of the approach is due to numerous reasonslike the computational simplicity and sparsity of the resulting model. A hugecollection of successful applications of LVQ can be found in [5].The basic LVQ approach relies on the major assumption that training dataand prototypes are uniquely associated with one specific class. This does notallow to represent uncertainty or possibility in the data. However, this is acommon phenomenon, e.g. in biological or medical applications [6]. In medicaldiagnosis for example, the relation of clinical and laboratory findings to dis-eases is probabilistic. Furthermore, input data can be imprecise due to missinginformation or additive noise.Extending Robust Soft Learning Vector Quantization (RSLVQ, [3]), we presenta novel LVQ variant which allows to realize multivariate class memberships fortraining data and prototypes. RSLVQ is based on a cost function which is de-fined in terms of a likelihood ratio. We derive the generalization of the RSLVQcost function with respect to vectorial class labels. In particular, we supposethat each element of the label vectors is in the range [0, 1] describing the prob-abilistic assignment of the data and the prototypes to a certain class. Training17ESANN 2011 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 27-29 April 2011, i6doc.com publ., ISBN 978-2-87419-044-5. Available from http://www.i6doc.com/en/livre/?GCOI=28001100817300.consists in the optimization of the extended cost function with respect to theprototype locations, the prototype labels and the algorithm’s hyperparameter(see [3]). In the working phase, classification is based on the value of the likeli-hood ratio, since this approach directly follows the objective of learning. As willbe demonstrated in the experiments, adaptive prototype labeling additionallysimplifies the model selection for LVQ systems.2 Review of Robust Soft LVQAssume training data {ξk, yk}lk=1 ∈ Rn × {1, . . . , C} are given, n denoting thedata dimensionality and C the number of different classes. An LVQ networkW = {(wj , c(wj)) : Rn × {1, . . . , C}}mj=1 consists of a number of n-dimensionalprototypes wj which are characterized by their location in feature space andtheir class label c(wj). Given a distance measure d(ξ,w) in Rn, classification isbased on a winner-takes-all scheme: a data point ξ ∈ Rn is assigned to the labelc(wi) of prototype wi with d(ξ,wi) ≤ d(ξ,wj), ∀j 6= i. Many LVQ variantsuse the squared Euclidean distance d(ξ,w) = (ξ −w)T (ξ −w) to quantify thesimilarity between feature vectors and the prototypes.The Robust Soft LVQ - algorithm [3] to train the prototype locations opti-mizes a cost function to adapt the prototypes to the data. The cost function isbased on a statistical modeling of the given data distribution, i.e. the probabilitydensity is described by a mixture model. It is assumed that every component jof the mixture generates data which belongs to only one of the C classes. Theprobability density of the data is approximated byp(ξ|W) =C∑i=1∑j:c(wj)=iP (j) p(ξ|j), (1)where∑j P (j) = 1, and the conditional density p(ξ|j) is a function of the pro-totype wj . A possible choice is the normalized exponential form p(ξ|j) = K(j) ·exp f(ξ,wj , σ2j ). In [3], a Gaussian mixture is assumed with K(j) = (2πσ2j )−n/2and f(ξ,wj , σ2j ) = −d(ξ,wj)/2σ2j ; d is the squared Euclidean distance, andevery component is assumed to have equal variance σ2j = σ2 and equal priorprobability P (j) = 1/m, ∀j. Accordingly, class-specific densities correspond top(ξ, y|W) =∑j:c(wj)=yP (j) p(ξ|j). (2)RSLVQ maximizes the likelihood ratioL =l∏k=1L(ξk, yk),with L(ξk, yk) =p(ξk, yk|W)p(ξk|W). (3)The RSLVQ cost function is defined as ERSLVQ = log (L), and it is maximizedwith respect to the prototype locations by means of stochastic gradient ascent.The learning rule is derived in [3]:∆wj =α1σ2{(Py(j|ξ)− P (j|ξ))(ξ −wj), c(wj) = y,−P (j|ξ)(ξ −wj), c(wj) 6= y,(4)18ESANN 2011 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 27-29 April 2011, i6doc.com publ., ISBN 978-2-87419-044-5. Available from http://www.i6doc.com/en/livre/?GCOI=28001100817300.−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−1−0.8−0.6−0.4−0.200.20.40.60.81ξPy(j|ξ)-P(j|ξ)  y1 =1, y2 =0y1 =0.9, y2 =0.1y1 =0.7, y2 =0.3−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−1−0.8−0.6−0.4−0.200.20.40.60.81ξPy(j|ξ)-P(j|ξ)  y1 =0, y2 =1y1 =0.1, y2 =0.9y1 =0.3, y2 =0.7Fig. 1: Illustration of the update strength affecting prototype w1 in a one-dimensionalsetting with prototypes w1 = −1, c1 = [0.8, 0.2] and w2 = 1, c2 = [0.4, 0.6]. Thecurves correspond to training samples with different label y. Left: y1 > 0.5, σ2 = 0.5.Right: y1 < 0.5, σ2 = 0.15.where α1 is the learning rate, and Py(j|ξ), P (j|ξ) are assignment probabilitiesPy(j|ξ) =exp f(ξ,wj , σ2)∑i:c(wi)=y exp f(ξ,wi, σ2), P (j|ξ) =exp f(ξ,wj , σ2)∑i exp f(ξ,wi, σ2)(5)with respect to one sample (ξ, y). Prototypes with c(w) = y are attracted bythe training pattern, while prototypes carrying any other class label are repelled.The training dynamics of RSLVQ highly depend on the hyperparameter σ2.Since it controls the assignment probabilities in Eq. (5), it controls the strengthof the attractive and repulsive forces in Eq. (4). In the limit σ2 → 0, RSLVQreduces to a learning-from-mistakes scheme (see [3] for details).3 Probabilistic LVQIn the following, we introduce the generalization of the RSLVQ cost functionwith respect to vectorial class labels for the prototypes and the input data. Weprovide the learning rules for the prototype vectors, the prototype labels andthe hyperparameter σ2 derived as a stochastic gradient ascent of the generalizedcost function. In the limit of crisp class memberships, the cost function and theupdates are equivalent to the original RSLVQ algorithm.Generalized cost function: In the generalized version of RSLVQ, input pat-terns and prototypes carry probabilistic label vectors y, c ∈ [0, 1]C , with∑i yi =1 and∑i ci = 1. Component yi constitutes the assignment probability of sam-ple ξ to class i; the same holds for the prototypes respectively. Under thisassumption, the definition of W (see Sec. 2) changes toW ={(wj , cj) |Rn × [0, 1]C ,with∑icij = 1}mj=1. (6)Accordingly, the global data density is defined byp(ξ|W) =C∑k=1m∑j=1ckj P (j) p(ξ|j). (7)19ESANN 2011 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 27-29 April 2011, i6doc.com publ., ISBN 978-2-87419-044-5. Available from http://www.i6doc.com/en/livre/?GCOI=28001100817300.The generalization of the class-specific density in Eq. (2) constitutes the weightedsum of the class-specific densities; the weight values are given by the assignmentprobabilities of the input sample to the different classesp(ξ,y|W) =C∑k=1ykm∑j=1ckj P (j) p(ξ|j). (8)The cost function of the generalized version of RSLVQ arises out of ERSLVQ bydefining the likelihood ratio (Eq. (3)) in terms of p(ξ|W) and p(ξ,y|W) as de-fined in Eq.s (7), (8). We term the new algorithm Probabilistic LVQ (PLVQ).Learning rules: In this contribution, we restrict our analysis to the adaptationof a global hyperparameter σ2 = σ2j , ∀j. To be in accordance with [3], we choosef(ξ,wj , σ2) and K(j) corresponding to a Gaussian mixture model and use thesquared Euclidean distance. Details of the derivatives ∂EPLVQ/∂{w,ξ,σ2} will bepresented elsewhere. The derivatives yield the following update rules for theprototypes and the hyperparameter∆wj =α1σ2Pj(ξ,y) (ξ −wj), ∆σ2 =α2σ2Pj(ξ,y)d(ξ,wj)σ2. (9)with Pj(ξ,y) =(Py(j|ξ)− P (j|ξ)), and Py(j|ξ) is the generalized class-specificassignment probabilityPy(j|ξ) =∑k ykckjP (j)K(j) exp f(ξ,wj , σ2)p(ξ,y|W). (10)The learning rule for the prototype labels results in∆cj = α3(yp(ξ,y|W)−1p(ξ|W))P (j) p(ξ|j). (11)The Probabilistic LVQ algorithm is defined in terms of Eqs. (9) and (11), andα1,2,3 are the learning rates.Note that Pj(ξ,y) turns into the weight factors in Eq. (4), if y and cjspecify crisp class memberships. Fig. 1 demonstrates the influence of y, cj andthe hyperparameter σ2 on Pj(ξ,y) in a one-dimensional example setting. In thisscenario, all training samples with non-crisp label cause attractive forces on w1for ξ → −∞(Pj(ξ,y) > 0) and have repulsive effects for ξ → +∞, (Pj(ξ,y) < 0).Contrary, training samples with crisp labeling have either attractive or repulsiveeffects all over feature space. The extreme values of the weight factors aredetermined by the prototype label c1 and c2; the hyperparameter σ2 controlsthe transition between attractive and repulsive effects.Classification: The original LVQ approach of distance-based classification isnot applicable in PLVQ, since the prototypes represent multiple classes. We pro-pose to use the value of the likelihood ratio defining the PLVQ cost function asclassification criterion for the new algorithm: An unknown feature vector ξ is as-signed to class i with L(ξ, i) > L(ξ, j), ∀j 6= i, where L(ξ, i) =p(ξ,yi|W)p(ξ|W)and thelabel vector yi specifies the unique assignment to class i. Highest likelihood-ratioclassification directly follows the objective of learning and has already shownpromising results in RSLVQ (see [7]).20ESANN 2011 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 27-29 April 2011, i6doc.com publ., ISBN 978-2-87419-044-5. Available from http://www.i6doc.com/en/livre/?GCOI=28001100817300.0 0.2 0.4 0.6 0.8 100.20.40.60.81w1w2w3  Class 1Class 2Class 30 0.2 0.4 0.6 0.8 100.20.40.60.81w1w2 w3w4w5  Class 1Class 2Class 3Fig. 2: Artificial data set. The figures show the prototype locations after PLVQ train-ing with three and five prototypes obtained in one training run. The correspondingprototype label are stated in Tab. 1.4 ExperimentsWe apply PLVQ to the artificial data visualized in Fig. 2. The data set consistsof three classes in a two-dimensional space. Each class consists of 60 sampleswhich are separated in two clusters. Note that we consider the special case oftraining data with crisp class memberships in this application. We compare theperformance of RSLVQ and PLVQ using different numbers of prototypes in com-bination with an adaptive hyperparameter σ2. We use the learning parametersettings α1,3 = 5 · 10−4 and α2 = σ2(0) · 10−5 with σ2(0) = 0.02. We randomlyselect 80% of the samples of each class for training and use the remaining datafor testing. We perform the experiment on 25 random constellations of trainingand test set and train the system for 200 epochs in each run. Prototypes areinitialized close the global mean value of the training data. Initial prototypelabels in PLVQ are generated randomly, followed by a normalization step.The mean error rates on the test sets are stated in Tab. 1, right. Due tothe nature of the data set, RSLVQ fails, if the data is approximated by only oneprototype per class. Remarkably, PLVQ already performs significantly betterwith the same number of prototypes. Fig. 2 (left) exemplary shows the finalTable 1: Artificial data set. Left: Prototype label after PLVQ-training with three andfive prototypes obtained in one training run. The corresponding prototype locationsare displayed in Fig. 2. Right: Mean test errors of RSLVQ and PLVQ using differentnumbers of prototypes.3 Prototypes 5 Prototypesc1 [0.5, 0.5, 0] [0.13, 0.87, 0]c2 [0.0, 0.0, 1.0] [0.0, 0.0, 1.0]c3 [0.0, 0.0, 1.0] [0.84, 0.16, 0.0]c4 [0.0, 0.0, 1.0]c5 [0.5, 0.5, 0.0]#Prototypes RSLVQ PLVQ3 0.5 0.354 0.315 0.286 0.18 0.2621ESANN 2011 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 27-29 April 2011, i6doc.com publ., ISBN 978-2-87419-044-5. Available from http://www.i6doc.com/en/livre/?GCOI=28001100817300.prototype constellations obtained in one experiment; the corresponding proto-type labels are stated in Tab. 1, left. Obviously, prototype w1 represents classesone and two equally. Note that increasing the number of prototypes in PLVQdoes not require the specification of the prototypes’ class memberships beforetraining is initialized. RSLVQ with two prototypes per class outperforms PLVQwith six prototypes. Note however that for this result prior information aboutthe structure of the data was taken into account to select the model settings.Another showcase of a PLVQ system consisting of five prototypes is given inFig. 2, right, and Tab. 1, left. It verifies that PLVQ detects prototypes withshared class memberships, if not all clusters can be approximated by individualprototypes.First experiments on high-dimensional real life data including probabilisticlabeling of training data showed promising results with local adaptive hyperpa-rameters σ2j . The adaptation of a global σ2 in PLVQ turned out to be insufficientin more complex classification scenarios. Detailed experiments and investigationsof this issue will be presented in forthcoming publications.5 ConclusionIn this contribution, we introduced a novel LVQ variant which allows to in-corporate probabilistic label information into the training process of the modelparameters. Probabilistic Learning Vector Quantization generalizes the success-ful Robust Soft LVQ such that vectorial labeling for prototypes and trainingsamples can be processed. In particular, the approach allows to realize adaptiveclass labels for the prototypes. It was demonstrated in experiments that thisability is especially beneficial, if no prior information about the structure of thedata is available.Two extensions of PLVQ will be subject to future work: As outlined in Sec. 4,we will investigate PLVQ with local, adaptive hyperparameters in real life appli-cations. Furthermore, a serious restriction of the algorithm consists in the use ofthe Euclidean distance measure. Metric adaptation techniques have shown greatpotential to improve the performance of RSLVQ [4], and comparable influencecan be expected for PLVQ.References[1] T. Kohonen. Self-Organizing Maps. Springer, Berlin, Heidelberg, second edition, 1997.[2] A. Sato and K. Yamada. Generalized learning vector quantization. In M. C. MozerD. S. Touretzky and M. E. Hasselmo, editors, Advances in Neural Information ProcessingSystems 8. Proceedings of the 1995 Conference, pages 423–9, Cambridge, MA, USA, 1996.MIT Press.[3] Sambu Seo and Klaus Obermayer. Soft learning vector quantization. Neural Computation,15(7):1589–1604, 2003.[4] P. Schneider, M. Biehl, and B. Hammer. Distance learning in discriminative vector quan-tization. Neural Computation, 21(10):2942–2969, 2009.[5] Bibliography on the Self-Organizing Map (SOM) and Learning Vector Quantization(LVQ). Neural Networks Research Centre, Helskinki University of Technology, 2002.[6] A. Torres and J.J. Nieto. Fuzzy logic in medicine and bioinformatics. Journal ofBiomedicine and Biotechnology, 2006(2), 2006.[7] P. Schneider, M. Biehl, and B. Hammer. Hyperparameter learning in probabilisitcprototype-based models. Neurocomputing, 73(7-9):1117–1124, 2010.22ESANN 2011 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 27-29 April 2011, i6doc.com publ., ISBN 978-2-87419-044-5. Available from http://www.i6doc.com/en/livre/?GCOI=28001100817300.

Multivariate class labeling in Robust Soft LVQ

https://pure.rug.nl/ws/files/41826345/es2011_66.pdf

Multivariate class labeling in Robust Soft LVQ

Abstract

Similar works

Full text

Available Versions

ARTS repository - University of Groningen