Biehl, M.

Gisbrecht, A.

Hammer, B.

Lueks, W.

Mokbel, B.

University of Groningen

   University of GroningenVisualizing the quality of dimensionality reductionMokbel, B.; Lueks, W.; Gisbrecht, A.; Biehl, M.; Hammer, B.Published in:20th European Symposium on Artificial Neural Networks ESANN 2012IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2012Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Mokbel, B., Lueks, W., Gisbrecht, A., Biehl, M., & Hammer, B. (2012). Visualizing the quality ofdimensionality reduction. In M. Verleysen (Ed.), 20th European Symposium on Artificial Neural NetworksESANN 2012 (pp. 179-184). d-side publishing.CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 12-11-2019Visualizing the qualityof dimensionality reductionBassam Mokbel1, Wouter Lueks2∗,Andrej Gisbrecht1, Michael Biehl2, Barbara Hammer11)Bielefeld University - CITEC Centre of Excellence, Germany2)University of Groningen - Faculty of Mathematics and Natural Sciences, NetherlandsAbstract. Many diﬀerent evaluation measures for dimensionality re-duction can be summarized based on the co-ranking framework [6]. Here,we extend this framework in two ways: (i) we show that the current pa-rameterization of the quality shows unpredictable behavior, even in simplesettings, and we propose a diﬀerent parameterization which yields more in-tuitive results; (ii) we propose how to link the quality to point-wise qualitymeasures which can directly be integrated into the visualization.1 IntroductionWith more and more nonlinear dimensionality reduction (DR) techniques becom-ing readily available, there is an increasing need for formal evaluation measuresto compare DR techniques, to optimize their parameters and solutions in case ofmultiple optima, and to directly assess the quality of such mappings. In recentyears, several quality measures have been proposed [4, 6, 11], many of which canbe summarized based on the co-ranking framework as proposed in [6]. In thiscontribution, we argue that the co-ranking framework covers even more casesincluding, e.g., DR evaluation based on information retrieval principles [11].The co-ranking framework in its current form, however, has a serious draw-back: even if data and its projections can be directly inspected, it is diﬃcultto predict the characteristics of the quality measure depending on its parameterK, the neighborhood size. The evaluation is typically subsumed in a qualitycurve over all possible K. In several simple scenarios, we demonstrate that itsshape is hard to predict. We argue that this behavior comes from small or con-tained errors in the mapping resulting in changes at a large range of K. Basedon this observation, we propose a diﬀerent parameterization which yields morepredictable behavior and which can directly highlight structurally interestingneighborhood sizes. By linking global quality measures to point-wise quantities,this observation gives rise to an integration of the quality into the visualizationitself, similar to heuristic alternatives in topographic mapping such as [1, 8].2 Evaluation measures for dimensionality reductionDR techniques are used to map a high-dimensional dataset Ξ = {ξ1, . . . , ξN} ⊂RD to a low-dimensional dataset X = {x1, . . . , xN} ⊂ RL, with L < D andL = 2 or L = 3 for the purpose of visualization. Let δij be the distance fromξi to ξj in RD and dij the distance from xi to xj in the low-dimensional space.The rank of ξj with respect to ξi in RD is given byρij = |{k | δik < δij or (δik = δij and 1 ≤ k < j ≤ N)}| .∗also aﬃliated with University of Nijmegen - Faculty of Science, The Netherlands179ESANN 2012 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 25-27 April 2012, i6doc.com publ., ISBN 978-2-87419-049-0. Available from http://www.i6doc.com/en/livre/?GCOI=28001100967420.Analogously, the rank of xj with respect to xi in the low-dimensional space isrij = |{k | dik < dij or (dik = dij and 1 ≤ k < j ≤ N)}| .The co-ranking matrix Q [6] is deﬁned by Qkl = |{(i, j) | ρij = k and rij = l}|.Errors of a DR mapping correspond to oﬀ-diagonal entries in this matrix. Apoint j where ρij > rij is an intrusion, if ρij < rij it is an extrusion. Usually, aDR mapping is not used to map all relationships of data faithfully. Often, thefocus is on the preservation of local relationships. The co-ranking matrix oﬀersa framework, in which several existing evaluation measures can be expressed,as pointed out in [6]: Local Continuity Meta Criterion (LCMC) [3], Trustwor-thiness & Continuity (T&C) [10], and Mean Relative Rank Errors (MRRE) [5].Essentially, these quality measures correspond to weighted sums of entries Qklof the co-ranking matrix for regions k, l ≤ K and ﬁxed neighborhood range K.In [6], an intuitive (unweighted) sum has been proposed, the Quality QNX:QNX(K) =1KNK∑k=1K∑l=1Qkl. (1)This summarizes all ‘benevolent’ points which change their rank only withina ﬁxed neighborhood K. To display the quality, usually the curve QNX(K) isplotted for a range K > 1. At present, no generally accepted transformation ofthis curve into a single meaningful value exists. In [7], an aggregated measureis proposed which averages the quality for values K ≤ k0 where the splittingpoint k0 is taken based on the distance of QNX to the baseline – however, thisautomatic choice is yet very sensitive to minor changes in the data.Interestingly, the quality measure QNX is identical to the Quality of PointNeighborhood Preservation (QNP) as proposed in [4]. Further, it coincides withthe measures Precision and Recall from [11] based on an information retrievalperspective, provided a local neighborhood of a point is deﬁned by its K-nearestneighbors in the original and the projection space, respectively.Distinct alternatives to the co-ranking framework are oﬀered by measure-ments which rely on -neighborhoods instead of K-nearest neighbors [11], onglobal correlations of the distances [4], or the distances of K-neighbors such asin the topographic product [2]. Note that the latter two take a more globalperspective, judging the global topology preservation rather than local rankings.3 Intuitive parameterization of the qualityMany diﬀerent quality measures are essentially similar to the quality QNX bytaking a weighted sum over entries Qkl, k, l ≤ K of the co-ranking matrix. Evenin very simple settings, however, it is diﬃcult to interpret the shape of the qualitycurve QNX(K). We consider three simple scenarios. The original data consist ofthree well separated Gaussian clusters in the plane containing 100 points each,see Fig. 1a. As a ‘mapping’, we consider the points in two dimensions obtainedby (i) a random swapping of points within every cluster only, (ii) a switch of thetwo leftmost clusters, (iii) the middle and leftmost cluster are stacked on top ofeach other. These simple artiﬁcial mappings represent typical behavior of DRmappings since they capture (i) local distortions, (ii) a tearing of regions, (iii)and an overlay of regions, common eﬀects of a too small projection dimension.180ESANN 2012 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 25-27 April 2012, i6doc.com publ., ISBN 978-2-87419-049-0. Available from http://www.i6doc.com/en/livre/?GCOI=28001100967420.−2.5 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 2.5−0.4−0.200.20.4  (a) original clusters50 100 150 200 250 30000.20.40.60.81Neighborhood sizes K  QNXQND(b) shuﬄed within clusters50 100 150 200 250 3000.50.60.70.80.91Neighborhood sizes K  QNXQND(c) left two clusters switched50 100 150 200 250 3000.50.60.70.80.91Neighborhood sizes K  QNXQND(d) left two clusters mergedFig. 1: Quality evaluation on artiﬁcial data to demonstrate the interpretabilityof the curves QNX vs. QND , for all settings of K.The resulting curves QNX are depicted in Fig. 1. Although we know the exactbehavior of the mapping, it is not easy to link the shape of QNX to structures ofthe mapping. Setting (i) yields local errors within the clusters of size 100 only,still QNX is below the optimum 1 for all neighborhood sizes K, and it suggestsa (not existing) structural match at size 200 by a local maximum of the curve,see Fig. 1b. A diﬀerent parameterization of the quality which we will introducebelow, QND, yields the alternative as depicted in Fig. 1b. Here, it is clearlyvisible that local distortions take place up to a neighborhood size 100 (withinclusters), while the mapping is almost perfect afterwards. The quality QNXsuggests perfect mappings for small neighborhood sizes up to 100 for setting (ii)and a suboptimal mapping for all larger scales, manifested in a severe drop of thequality for K > 100, see Fig. 1c. In contrast, the measure QND indicates a slightdrop of accuracy for the range K ∈ {100, . . . , 200} which exactly corresponds tothe region where the two leftmost clusters are switched. Similarly, QNX depictsa mapping quality which is not optimum for all ranks K for case (iii) althoughonly the leftmost two clusters are stacked on top of each other, hence signiﬁcantdistortions take place in the range K ∈ {1, . . . , 100} only. In contrast, this factis clearly indicated when regarding QND, see Fig. 1d.What is the reason behind these eﬀects? QNX uses the neighborhood size Kfor two diﬀerent purposes: on the one hand, K singles out the region of interestby determining the size of the neighborhood of points taken into account, namelyρij ≤ K. On the other hand, it determines the size and shape of errors whichare tolerated: rij ≤ K is not counted as an error in the region of interest.Note that the actual size of the rank error is not considered at all as long asthe rank is smaller than K. This parameterization has the eﬀect that smallrank errors can contribute to the shape of QNX(K) for every value K. Considerthe following extreme setting: Points are swapped pairwise as shown in Fig. 2.Obviously, rank errors of size at most 4 occur due to the only local changes.181ESANN 2012 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 25-27 April 2012, i6doc.com publ., ISBN 978-2-87419-049-0. Available from http://www.i6doc.com/en/livre/?GCOI=28001100967420.This is clearly mirrored by the co-ranking matrix displayed in Fig. 2 for whichonly four oﬀ-diagonals are not equal to 0. Due to the choice of QNX, however,the quality does not approach one, rather errors occur for every neighborhoodK, see Fig. 2c. This property is shared by any quality measure which relies onthe same part Qkl, k, l ≤ K of the co-ranking matrix.This observation also suggests how to change the parameterization of QNXwith respect to the co-ranking matrix to obtain more intuitive results, as depictedin Fig. 3. As a ﬁrst step, instead of taking a single parameter K, one candecouple the control parameter for the region of interest Ks and the controlparameter for the tolerated errors Kt, thus summing over a rectangular partQkl, k ≤ Ks, l ≤ Kt, see Fig. 3b. As a second step, one can choose the errortolerance based on the actual size of rank errors instead of accepting all ranksat most Kt, i.e. we sum over κt oﬀ-diagonals instead of the ﬁrst Kt columns,see Fig. 3c. This choice yields a quality measure parameterized by the region ofinterest Ks and the tolerated rank error κt:QND(Ks, κt) =1KsN∑i≤Ks∑j:|i−j|≤κtQij .To reduce the number of parameters and hence the complexity, we suggestto choose the tolerated error to be of the same size as the region of interest:κt = Ks =: κ, resulting in a reparameterized quality where, similar to [6], nor-malization takes place to guarantee values in the interval [0, 1]:QND(κ) =1κN∑i≤κ∑j:|i−j|≤κQij .Unlike QNX, the measure QND makes sure that small rank errors contribute onlyto values QND(K) for small K and they do not spread over the entire qualitycurve, making the behavior more predictable.b a d c f e h ga b c d e f g h(a) Switching of points                                                                                                               (b) Q5 10 15 200.50.60.70.80.91  QNXQND(c) QNX(K)Fig. 2: The left shows a simple switching scheme of one-dimensional points.Obviously, rank errors are at most four (in case of tie breaks) in this setting.This is mirrored by the shape of the co-ranking matrix (in the middle for thesame setting with 20 points) for which four oﬀ-diagonals are non-vanishing, andby QND(K)=1 for all K > 4, while QNX(K) < 1 for almost all K (on the right).182ESANN 2012 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 25-27 April 2012, i6doc.com publ., ISBN 978-2-87419-049-0. Available from http://www.i6doc.com/en/livre/?GCOI=28001100967420.KK(a) originalKtKs(b) rectangularκtκtKs(c) w.r.t. to diagonalκκ(d) Ks=κt w.r.t. diag.Fig. 3: Change of the summation area of the co-ranking matrix to better controlthe region of interest and the tolerated rank errors.4 Point-wise quality measureIt has been pointed out e.g. in the work [1, 8] that a given visualization shouldalso provide a direct intuition in how far it can be relied on. Ideally, together witha mapping of points, the quality of a mapped point should be visualized directlyat its respective location. While the approaches [1, 8] provide very eﬀectiveheuristics to do so, surprisingly, none of the formal DR evaluation measureshave been used so far to directly visualize the quality of the projections.Note that the co-ranking matrix can easily be accompanied by a correspond-ing point-wise framework: Qikl = |{j|ρij = k and rij = l}| which gives rise topoint-wise quality measures QiNX(K) =∑k≤K∑l≤K Qikl/K and QiND(K) =∑k≤K∑l:|k−l|≤K Qikl/K which, averaged over all points, again yield the qualitymeasures QNX and QND, respectively. These measures can be used to display,together with the DR, the quality of the projection at every point by means ofa color value corresponding to QiNX(K) or QiND(K) for relevant K. Thereby, Kcan be chosen according to relevant structural criteria such as a local extremumof the curve QNX or QND, or it can be chosen interactively according to the usersneeds.An example is given in Fig. 4 for the classical swiss roll data set visualizedusing t-SNE [9] and choosing K according to the ﬁrst local optimum of the qual-ity curve. Interestingly, the measure QiND(K) clearly singles out positions wheretearing and topological mismatches occur, while QiNX(K) shares the problem ofQNX that local rank errors can aﬀect diﬀerent regions of the curve, thus alsoindicating many less pronounced points in the mapping.5 ConclusionsWe have discussed quality measures based on the co-ranking framework andproblems which arise if quadratic regions of the matrix are used to deﬁne aquality measure. In contrast, we have proposed an alternative parameterizationscheme which yields more intuitive behavior, and by linking the co-ranking ma-trix to a point-wise version, which also oﬀers meaningful local color values tovisualize the quality of a given mapping. Note that it may be worthwhile to alsostudy the behavior of the quality matrix QND(Ks, κt) depending on both pa-183ESANN 2012 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 25-27 April 2012, i6doc.com publ., ISBN 978-2-87419-049-0. Available from http://www.i6doc.com/en/livre/?GCOI=28001100967420.  worstbest  worstbestFig. 4: A mapping of the swiss roll data by t-SNE (with a perplexity of 50),colored by point-wise qualities QiNX(14) (left) and QiND(14) (right). The coloringclearly indicates the tearing of the original manifold, with less ambiguity in QiNDon the right. (The sequence of class labels from the inside to the outside of theoriginal spiral-shaped manifold is: ◦.)rameters, the relevant range Ks and the tolerated rank errors κt, since it allowsusers to interactively choose appropriate values according to a given application.References[1] M. Aupetit. Visualizing distortions and recovering topology in continuous projectiontechniques. Neurocomputing, 70(7-9):1304 – 1330, 2007.[2] H.-U. Bauer, K. Pawelzik, and T. Geisel. A topographic product for the optimization ofself-organizing feature maps. In NIPS, pages 1141–1147, 1991.[3] L. Chen and A. Buja. Local multidimensional scaling for nonlinear dimension reduction,graph drawing, and proximity analysis. Journal of the American Statistical Association,104(485):209–219, 2009.[4] A.N. Gorban and A.Y. Zinovyev. Principal manifolds and graphs in practice: from molec-ular biology to dynamical systems. Int. J. Neural Syst., 20(3):219–232, 2010.[5] J.A. Lee and M. Verleysen. Nonlinear dimensionality reduction. Springer, 2007.[6] J.A. Lee and M. Verleysen. Quality assessment of dimensionality reduction: Rank-basedcriteria. Neurocomput., 72(7-9):1431–1443, 2009.[7] J.A. Lee and M. Verleysen. Scale-independent quality criteria for dimensionality reduc-tion. Pattern Recognition Letters, 31:2248–2257, October 2010.[8] A. Ultsch and H.P. Siemon. Kohonen’s self organizing feature maps for exploratory dataanalysis. In Proceedings of INNC’90, pages 305–308. Kluwer, 1990.[9] L.J.P. van der Maaten and G.E. Hinton. Visualizing high-dimensional data using t-sne.Journal of Machine Learning Research, 9:2579–2605, 2008.[10] J. Venna and S. Kaski. Local multidimensional scaling. Neural Netw., 19:889–899, 2006.[11] J. Venna, J. Peltonen, K. Nybo, H. Aidos, and S. Kaski. Information retrieval perspec-tive to nonlinear dimensionality reduction for data visualization. J. Mach. Learn. Res.,11:451–490, 2010.184ESANN 2012 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 25-27 April 2012, i6doc.com publ., ISBN 978-2-87419-049-0. Available from http://www.i6doc.com/en/livre/?GCOI=28001100967420.

Visualizing the quality of dimensionality reduction

https://pure.rug.nl/ws/files/41826209/es2012_99.pdf

Visualizing the quality of dimensionality reduction

Abstract

Similar works

Full text

Available Versions

University of Groningen