Copyright © 2005 University of ExeterWhen estimating how much better a classifier is than random allocation in Q-class ROC analysis, we need to sample from a particular region of the unit hypercube: specifically the region, in the unit hypercube, which lies between the Q − 1 simplex in
Q(Q − 1) space and the origin.
This report introduces a fast method for randomly sampling this volume, and is compared to rejection sampling of uniform draws from the unit hypercube. The new method is based on sampling from a Dirichlet distribution and shifting these samples using a draw from the Uniform distribution. We show that this method generates random samples within the volume at a probability ≈ 1/(Q(Q − 1)), as opposed to ≈ (Q − 1)Q(Q − 1) /(Q(Q − 1))! for rejection sampling from the unit hypercube.
The vast reduction in rejection rates of this method means comparing classifiers in a Q-class ROC framework is now feasible, even for large Q.Department of Computer Science, University of Exete

Fieldsend, Jonathan E.

Open Research Exeter

School of Engineering, Computer Scienceand Mathematics,University of Exeter. EX4 4QF. UK.http://www.ex.ac.uk/secsmA short note on the efficient random samplingof the multi-dimensional pyramidbetween a simplex and the originlying in the unit hypercubeJonathan E. FieldsendSchool of Engineering, Computer Science and Mathematics,University of Exeter, Exeter, EX4 4QF, UKJ.E.Fieldsend@exeter.ac.uk24th August 2005AbstractWhen estimating how much better a classifier is than random allocation in Q-class ROCanalysis, we need to sample from a particular region of the unit hypercube: specifically theregion, in the unit hypercube, which lies between the Q− 1 simplex in Q(Q− 1) space and theorigin.This report introduces a fast method for randomly sampling this volume, and is comparedto rejection sampling of uniform draws from the unit hypercube. The new method is basedon sampling from a Dirichlet distribution and shifting these samples using a draw from theUniform distribution. We show that this method generates random samples within the volumeat a probability ≈ 1/(Q(Q − 1)), as opposed to ≈ (Q − 1)Q(Q−1)/(Q(Q − 1))! for rejectionsampling from the unit hypercube.The vast reduction in rejection rates of this method means comparing classifiers in a Q-classROC framework is now feasible, even for large Q.1 IntroductionIn our recent work, we have introduced a method for extending the binary class Receiver OperatingCharacteristic (ROC) analysis (Hanley and McNeil [1982]; Zweig and Campbell [1993]) to the multi-class domain (Everson and Fieldsend [2005]; Fieldsend and Everson [2005]). Rather than consideringthe true and false positive rates, we considered the multi-class ROC surface to be the solution ofthe multi-objective optimisation problem in which these misclassification rates are simultaneouslyoptimised. Srinivasan [1999] has discussed a similar formulation of multi-class ROC, showing thatif classifiers for Q classes are considered to be points with coordinates given by their Q(Q − 1)misclassification rates, then optimal classifiers lie on the convex hull of these points.In the evaluation of this multi-class ROC analysis, we developed a straightforward generalisation ofthe Gini coefficient which quantified the superiority of a classifier’s performance to random allocation.In order to estimate this value, the need arises to generate uniformly distributed samples from aspecific region of the unit hypercube, namely the region corresponding to that lying between theQ− 1 simplex in Q(Q− 1) space and the origin, which defines classifier misclassification rate space10:48 24th August 20052 Comparing classifiers 2which is better than random allocation. The cost of sampling this region becomes prohibitivelylarge at even relatively small Q when rejection sampling from the unit hypercube is used. Here weintroduce a method of uniform sampling from the space which is vastly less expensive.2 Comparing classifiersIn two class problems the area under the ROC curve (AUC) is often used to compare classifiers. Asclearly explained by Hand and Till [2001], the AUC measures a classifier’s ability to separate twoclasses over the range of possible costs and is linearly related to the Gini coefficient.By analogy with the AUC, we can use the volume of the Q(Q − 1)-dimensional hypercube that isdominated by elements of the ROC surface for classifier A as a measure of A’s performance. Inbinary and multi-class problems alike its maximum value is 1 when A classifies perfectly. If theclassifier allocates at random, the ROC surface is the simplex in D = Q(Q − 1)-dimensional spacewith vertices at length Q − 1 along each coordinate vector. The volume of the unit hypercubedominated by this can be derived as follows: As the volume of the pyramidal region between theorigin and the simplex with vertices at a distance L along each coordinate vector is LDD! . The volumelying between the origin and the random allocation simplex is, therefore:(Q− 1)DD!. (1)Only part of this volume lies in the unit hypercube however, as the corners (excluding that at theorigin) relate to infeasible regions where classification rates are > 1. Each of these D corner regionsis also a pyramidal volume, but with sides of length Q− 2. The total volume of the region betweenthe origin and the random allocation simplex which also lies in the unit hypercube is therefore(Q− 1)DD!−D(Q− 2)DD!. (2)C21C120 11ABFigure 1. Illustration of the G and δ measures where Q = 2. Shaded area denotes G(A), horizontally dashedarea denotes δ(A,B), vertically dashed area denotes δ(B,A).We denote this region by P . When Q = 2 the second term in equation 2 is zero so that the totalvolume (area) between the origin and the random allocation simplex is just 1/2. This correspondsto the area under the diagonal in a conventional ROC plot (although binary ROC plots are usuallymade in terms of true positive rates versus false positive rates for one class, the false positive rate forthe other class is just 1 minus the true positive rate for the other class). However, when Q > 2, thevolume not dominated by the random allocation simplex is very small; even when Q = 3, the volumenot dominated is ≈ 0.0806. We therefore define G(A) to be the analogue of the Gini coefficient intwo dimensions, namely the proportion of the volume of the D-dimensional unit hypercube that isdominated by elements of the ROC surface, but is not dominated by the simplex defined by randomallocation (as illustrated by the shaded area in Figure 1 for the Q = 2 case). In binary classification2.1 Rejection sampling from the unit hypercube 3problems this corresponds to twice the area between the ROC curve and the diagonal. In multi-classproblems G(A) quantifies how much better A is than random allocation. It can be simply estimatedby Monte Carlo sampling of this volume in the unit hypercube.2.1 Rejection sampling from the unit hypercubeA simple manner to generate uniform samples, x, of this volume is to generate uniform samples fromthe unit hypercube x ∼ U(0, 1)Q(Q−1) and reject those values where∑xi ≥ Q − 1 or where anyelement xi > 1 (as shown in Algorithm 1). We know from Equation 2 the probability of generatingan acceptable sample, and Figure 2 shows this for Q = 2, . . . , 10.Algorithm 1 Uniform rejection sampling from the unit hypercube.Inputs:N Number of random samples from regionQ Number of classesX Set of N random samples from region1: n = 02: X = {}3: while n 6= N4: x := U(0, 1)Q(Q−1)5: if∑xi < Q− 1 ∧ xi ≤ 1 ∀i6: X := X ∪ x7: n := n+ 18: end9: end2 4 6 8 1010−6010−5010−4010−3010−2010−10100QProb. generating sampleFigure 2. The probability of generating a random value in the unit hypercube which is not dominated by thesimplex defined by random allocation.As can be seen, this value becomes vanishingly small as Q increases, meaning that rejection samplingfrom the hypercube becomes infeasible for even relatively small Q. This necessitates a less wastefulgeneration of random samples before Q-class ROC classifier comparison becomes computationallyfeasible. This can be achieved through the combined use of a Dirichlet and Uniform distribution,which we now discuss.2.2 Using a transformation of the Dirichlet 42.2 Using a transformation of the DirichletFirst let us concern ourselves with the generation of samples below the simplex. The Dirichletdistribution generates samples from the simplex where∑yi = 1p(y) = Dir(y |α1, . . . , αD) (3)=Γ(∑Di=1 αi)∏Di=1 Γ(αi)(1−D−1∑i=1yi)αD−1 D−1∏i=1yαi−1i (4)where the index i labels theD = Q(Q−1) elements. The αi ≥ 0 determine the density of the samples;by setting all the αi = 1 the simplex is sampled uniformly with respect to Lebesgue measure. Figure3 shows 10000 samples generated from the Dirichlet distribution where D = 2, 3.Algorithm 2 Random sampling from the region of interest using a combination of Dirichlet andUniform distributions.Inputs:N Number of random samples from regionQ Number of classesX Set of N random samples from region1 A Q(Q− 1)-dimensional vector of ones1: n = 02: X = {}3: while n 6= N4: y := Dir(1)5: z := U(0, 1Q(Q−1) )6: θ := (Q− 1)(y − z1)7: if θi ≤ 1 ∀i ∧ θi ≥ 0 ∀i8: X := X ∪ x9: n := n+ 110: end11: endIf we now generate a uniform sample z ∼ U(0, 1/D), then y− z1, where 1 is a D-dimensional vectorof 1s, creates a transformed sample, θ, uniformly distributed in a hyper-prism. Figure 4 showssamples generated for Figure 3 shifted in this fashion.A simple check of sign of the elements of θ then allows us to extract those lying in the pyramidbetween the origin and the simplex (as shown in Figure 5).In order to transform these points to ones lying beneath the Q− 1 simplex the elements are simplymultiplied by Q− 1. As this doesn’t affect the assignment to whether a point lies in the pyramid ornot, the illustrations using unit simplex (Figures 3-5) are general. Algorithm 2 steps through thisprocess0 0.2 0.4 0.6 0.8 100.10.20.30.40.50.60.70.80.9100.51 0 0.20.4 0.60.8 100.20.40.60.81Figure 3. 10000 samples generated from the Dirichlet distribution. Left: D = 2; Right: D = 3.2.2 Using a transformation of the Dirichlet 5−0.5 0 0.5 1−0.500.51−0.500.51−0.500.51−0.500.51Figure 4. 10000 samples generated from the Dirichlet distribution, shifted by the subtraction of uniformlydistributed values. Left: D = 2; Right: D = 3.0 0.2 0.4 0.6 0.8 100.10.20.30.40.50.60.70.80.9100.51 0 0.20.4 0.60.8 100.20.40.60.81Figure 5. Those samples generated from a Dirichlet distribution and shifted by the subtraction of uniformlydistributed values, which lie in the pyramid. Left: D = 2; Right: D = 3.3 Rejection rate of the proposed approach 63 Rejection rate of the proposed approachFor any given Q we can easily define the convex hull of the pyramidal region bounded by the simplexand the origin, and also the parallelapiped defined by the sampled volume of the simplex projection.3.1 Volume ratiosTable 1. Calculated volumes of regions for various Q.Q 2 3 4 5 6Pyramid volume 0.5 8.89× 10−2 1.11× 10−3 4.52× 10−7 3.51× 10−12(Q− 1)Q(Q−1)/(Q(Q− 1))! 0.5 8.89× 10−2 1.11× 10−3 4.52× 10−7 3.51× 10−12Sampled volume 1 5.33× 10−1 1.33× 10−2 9.04× 10−6 1.05× 10−10Volume ratio 2 6 12 20 30Table 1 shows the volumes of these two regions calculated for various Q. As can clearly be seen,the ratio of the two regions’ volumes is actually D. The proportion of samples generated in thistwo step fashion that lie in the pyramid between the origin and the simplex is therefore 1/D. Usingequation 2, the proportion of points generated in this fashion that lie between the Q − 1 simplexand the origin and also lie in the unit cube is:=1D((Q−1)DD! −D(Q−2)DD!)((Q−1)DD!) (5)=1D(1−D(Q− 2)D(Q− 1)D)(6)≈1Q(Q− 1)as Q becomes large. (7)These points can essentially be found from those lying in the pyramidal region by rejecting thosewith elements θi > 1. Figure 6 shows the probability of generating an acceptable sample using thismethod for Q = 2, . . . , 10.2 3 4 5 6 7 8 9 1010−210−1100QProb. generating sampleFigure 6. The probability of generating a random value in the unit hypercube which is not dominated bythe simplex defined by random allocation, using the method of combing samples from Dirichlet and Uniformdistributions.Note that the second term in (5) rapidly approaches one, so the proportion of samples generatedbetween the random allocation simplex and the origin that lie in the unit hypercube can be takenas ≈ 1/(Q(Q−4 Example 74 ExampleIf every point on the optimal ROC surface for classifier A is dominated by a point on the ROCsurface for classifier B, then classifier B has a superior performance to classifier A. In general,however, neither ROC surface will completely dominate the other: regions of A’s surface will bedominated by B and vice versa; in binary problems this corresponds to ROC curves that cross. Toquantify the classifier’s relative performance we therefore define δ(A,B) to be the volume of P thatis dominated by elements of A and not by elements of B (marked in Figure 1 with horizontal lines).Note that δ(A,B) is not a metric; although it is non-negative, it is not symmetric. Also if A and Bare subsets of the same non-dominated set W, (i.e., A ⊆W and B ⊆W ), then δ(A,B) and δ(B,A)may have a range of values depending on their precise composition; see Fieldsend et al. [2003] formore details. Situations like this are rare in practice, however, and measures like δ have proveduseful for comparing Pareto fronts.Table 2. Generalised Gini coefficients and exclusively dominated volume comparisons of the multinomial logisticregression (MLR) and k-nn classifiers. Taken from Fieldsend and Everson [2005].Measure Synth. Vehicle ImageG(MLR) 0.840 ≈ 0 ≈ 0G(k-nn) 0.920 0.168 0.076δ(MLR, k-nn) 0.001 0.000 0.000δ(k-nn,MLR) 0.081 0.168 0.076Table 2 shows the results of comparing the performance of the probabilistic k -nn classifier [Holmesand Adams, 2002] with that of the multinomial logistic regression classifier on Synthetic data, andalso on the UCI Image and Vehicle data sets [Blake and Merz, 1998]. These were calculated from 105uniform samples from the region between the random allocation simplex and the origin. Numericallythe values obtained using rejection sampling from the unit hypercube for the Synthetic and Vehicledata sets were the same as those presented here. The exorbitant computation cost of rejectionsampling for the Q(Q − 1) = 42 case of the Image dataset meant we did not compare the twotechniques for the final set.5 ConclusionA method has been introduced for the uniform sampling of the space between the random allocationsimplex and the origin for multi-class ROC assessment purposes. This method has been shown to beorders of magnitude more efficient than the simple rejection sampling of the unit hypercube – with itscomparable efficiency increasing with the number of classes. This method generates random sampleswithin the volume at a probability ≈ 1/(Q(Q− 1)), as opposed to ≈ (Q− 1)Q(Q−1)/(Q(Q− 1))! forrejection sampling from the unit hypercube.Matlab code to generate samples from this region is provided from the author’s website at http://www.dcs.ex.ac.uk/people/jefields/, along with code for proving the sample rejection rate.AcknowledgementsThe author would like to thank Richard Everson for his help and discussion in the generation of thisreport.ReferencesC.L. Blake and C.J. Merz. UCI repository of machine learning databases, 1998. URL http://www.ics.uci.edu/~mlearn/MLRepository.html.REFERENCES 8R.M. Everson and J.E. Fieldsend. Multi-class ROC analysis from a multi-objective optimisationperspective. Technical Report 421, Department of Computer Science, University of Exeter, April2005.J.E. Fieldsend and R.M. Everson. Formulation and comparison of multi-class ROC surfaces. In Pro-ceedings of the 2nd ROC Analysis in Machine Learning Workshop, part of the 22nd InternationalConference on Machine Learning (ICML 2005), pages 41–48, 2005.J.E. Fieldsend, R.M. Everson, and S. Singh. Using Unconstrained Elite Archives for Multi-ObjectiveOptimisation. IEEE Transactions on Evolutionary Computation, 7(3):305–323, 2003.D.J. Hand and R.J. Till. A simple generalisation of the area under the ROC curve for multiple classclassification problems. Machine Learning, 45:171–186, 2001.J.A. Hanley and B.J. McNeil. The meaning and use of the area under a receiver operating charac-teristic (ROC) curve. Radiology, 82(143):29–36, 1982.C.C. Holmes and N.M. Adams. A probabilistic nearest neighbour method for statistical patternrecognition. Journal Royal Statistical Society B, 64:1–12, 2002.A. Srinivasan. Note on the location of optimal classifiers in n-dimensional ROC space. TechnicalReport PRG-TR-2-99, Oxford University Computing Laboratory, Oxford, 1999. URL ftp://ftp.comlab.ox.ac.uk/pub/Packages/ILP/Papers/AS/roc.ps.gz.M.H. Zweig and G. Campbell. Receiver-operating characteristic (ROC) plots: a fundamental eval-uation tool in clinical medicine. Clinical Chemistry, 39:561–577, 1993.

A short note on the efficient random sampling of the multi-dimensional pyramid between a simplex and the origin lying in the unit hypercube

Abstract

Similar works

Full text

Available Versions

Open Research Exeter