We introduce the idea of Characteristic Regions to solve a classification problem. By identifying regions in which classes are dense (i.e. many observations) and also relevant (for discrimination) we can characterize the different classes. These Characteristic Regions are used to generate a classification rule. The result can be visualized so the user is provided with an insight into data for an easy interpretation

Luebke, Karsten

Szepannek, Gero

EconStor (ZBW Kiel)

Different Subspace Classification

Eldorado - Ressourcen aus und für Lehre, Studium und Forschung

Different Subspace ClassificationGero Szepannek ⋆ and Karsten LuebkeUniversity of Dortmund⋆⋆, Department of Statistics, 44221 Dortmund, GermanyAbstract. We introduce the idea of Characteristic Regions to solve a classi-fication problem. By identifying regions in which classes are dense (i.e. many ob-servations) and also relevant (for discrimination) we can characterize the differentclasses. These Characteristic Regions are used to generate a classification rule. Theresult can be visualized so the user is provided with an insight into data for an easyinterpretation.1 IntroductionSupervised Classification or Discrimination often involves two goals: the firstis allocation or prediction, i.e. assigning class labels to new observations. Thesecond goal, which can be even more important, is descriptive and involvesthe disclosure of the underlying differences between the classes. The newDifferent Subspace Classification (DiSCo) method is a method to simultane-ously visualize and classify multi-class-problems in high dimensional spacesand therefore is designed to attain both predictive and descriptive goals.The problem of classification or pattern recognition is given in the follow-ing way: N objects xn, n = 1, . . . , N , are observed, each object belonging toone and only one class kn, kn ∈ {1, . . . ,K}, n = 1, . . . , N . The class member-ship is known to the user. Nk objects are observed from class k. This set ofobjects is called training data. For each object D variables xd, d = 1, . . . , D,are observed. Every object xn can be considered as a D-dimensional realiza-tion of a random vector Xn following an unknown distribution that dependson its class kn.The first goal is to be able to determine the correct (unknown) class forobjects xnew that will be observed in future. The second goal is to find outthe characteristics of the different classes by analyzing the training data. Thehigher the dimension of the data the more challenging is the understandingof the data. So if there are many observed variables, methods of variableselection are often used to reduce the dimension of the data. These methodsidentify and retain those of the variables that separate the classes best. Thenfollowing this procedure a classification method is (re-)applied to the resultingsubspace of variables. A problem may be that in general the variables do notcontain equal separating-information for all classes. So a variable can contain⋆⋆ This work has been supported by the Deutsche Forschungsgemeinschaft, Sonder-forschungsbereich 475.⋆ e-mail: szepannek@statistik.uni-dortmund.de2 Szepannek and Luebkeinformation for separating class i from the rest but no information for theseparation of class j 6= i.In DiSCo variable selection is intrinsic to the classification method. Theresulting subsets of variables which are used for discrimination of the classescan differ between the classes.A focus is also laid on the visualization of the class-characteristics. The pro-posed method does not make any assumptions about the underlying distri-bution of the data. The only weak assumption is that objects of the sameclass are similar in some of their predictor values.In the following chapter the principle of Characteristic Regions is definedand a classification rule developed. Chapter 3 explains the visualization ofthe results. Chapter 4 briefly summarizes the choice of parameters for theimplementation of the method while chapter 5 contains a simulation studywith comparison to Classification trees and Discriminant analysis.2 Notation and MethodThe idea of the new method is to search for Characteristic Regions, i.e. setsof values in some variables that indicate the class-membership. To build upthese Characteristic Regions two steps are needed. The first step is to searchfor intervals of the realizations of the random variables that contain a largeprobability mass of the classes. The resulting ”regions” are called Dense Re-gions. The second step, which is independent of the first, identifies regionsthat discriminate at least one class from the others because of a relativelyhigh density. These regions are called Relevant Regions. Regions that areboth dense and relevant are then called Characteristic Regions.2.1 Characteristic RegionsDefinition 1. S being the set of all possible predictor values of an objectxn, for all d let {Rdm : 0 ≤ m ≤Md+1} be a contiguous segmentation of aninterval covering S ∩Xd following1.⋃Md+1m=0 Rdm ⊇ S ∩Xd(All possible values of Xd are covered by the union of all its regions.)2. ∀x1, x2 ∈ Rdm and α ∈ [0, 1] : αx1 + (1− α)x2 ∈ Rdm(The regions of every variable are contiguous.)3. ∀x1 ∈ Rdm1 , x2 ∈ Rdm2 , m1 < m2 : x1 < x2(In every variable the regions are disjoint and also ordered.)Rdm are called regions of variable Xd.By restriction 2 all the objects that fall into one region can be considered tobe similar.DiSCo - Different Subspace Classification 3Definition 2. Let xdn be the value taken by object n in variable Xd and letkn be the corresponding, known index of its class. Thenndm(k) :=N∑n=1I[Rdm](xdn) I[k](kn) (1)with I[·] as the indicator function is called the corresponding frequency ofclass k in Region m of variable d.As the ndm(k) should represent the density of the data it is assumedfor simplicity of comparisons that for any fixed d and all 1 ≤ m ≤ Md :supx∈Rdm − infx∈Rdm ≡ const., so the regions of a variable have equal width.By this the corresponding frequencies are proportional to heights of histogrambars of the classes if the bandwidths are given by the regions.Let Dense Regions be those regions which contain most of the classes’probability masses. Let SDR > 0 be a threshold to construct classwise DenseRegions. Then Dense Regions are regions Rdm0(k) withndm0(k) ≥ SDR∑Md+1m=0 ndm(k)Md(2)This proceeding corresponds to comparing the observed corresponding fre-quency to the mean over all regions.Relevant Regions should be the regions where the density of one classk is high compared to those of the other classes and so a new observed objectlying in this region strongly indicates its membership to class k. Let SRR >0 be a threshold to construct classwise Relevant Regions. Then RelevantRegions are regions Rdm(k0) with:ndm(k0)Nk0≥ SRR∑Kk=1ndm(k)NkK(3)To be able to compare the regions’ densities of different classes by correspond-ing frequencies they have to be weighted by their observed absolute frequen-cies. Finally, Characteristic Regions are regions that are both dense andrelevant.2.2 Classification RuleLet wdm(k) ≥ 0 be the class wise weight of a region of class k connectedto region Rdm.The Characteristic Regions are used to build up the classification rule bysumming the weights over all variables. Then the assignment of the class isobtained bykˆ(xnew) = argmaxkD∑d=1Md+1∑m=0I[Rdm](xnew)wdm(k) (4)4 Szepannek and Luebkewhere the weights of the Characteristic Regions are defined bywdm(k0) :=0 if (2) or (3) do not holdndm(k0)p(k0)NNk0∑Kk=1 ndm(k)p(k)NNkif Rdm is characteristic for class k0(5)p(k)NNkis a correction term for the absolute frequency of the classes in thedata with the prior probabilities of the classes – if it differs from the observedfrequency. The weights are motivated by the marginal probability of knew = kgiven xdnew ∈ Rdm, if Rdm is ”characteristic” for class k.As only Characteristic Regions are used for the classification rule thecutpoints of the regions may disregard information. So to keep more of theclasses’ probability masses we propose another smoothed classification rulewhere the weights wdm(k) are as before but additionally the adjoining regionsare included in the model. Then:kˆ(xnew) = argmaxkD∑d=1Md+1∑m=0I[Rdm](xnew)(12wdm−1(k) + wdm(k) +12wdm+1(k))(6)with wdm(k) = 0 for m = −1,Md + 2.3 VisualizationThe weights wdm(k) described above mimic marginal conditional probabilityof the different classes. As only Characteristic Regions will be shown in ourvisualization only robust information relevant for classification is given. Soplotting these class wise weights of the regions (see equation 5) provides avisualization of the class characteristics and an interpretation may be sim-plified.As example we illustrate the method in Figure 1 on the well known Irisdata set introduced by Fisher. The values of the variables are shown on the x-axes while the different colours of the bars symbolize the different true classes(black = ”Setosa”, light grey = ”Virginica” and dark grey = ”Versicolor”).The heights are the weights of the Characteristic Regions. It can be seenthat the variable ”Sepal length” only serves to indicate membership of oneof the classes ”Virginica” or ”Setosa” but not for ”Versicolor”, while thevariable ”Sepal width” just serves to characterize a plant of class ”Setosa” or”Versicolor”.The ”Petal” variables seem to separate all three classes with the lowestvalues for class ”Setosa”.The upper extreme values indicate the class ”Vir-ginica”. As the plots of these two variables are of the same structure one cansuppose a correlation between these variables.DiSCo - Different Subspace Classification 54.3 4.9 5.5 6.1 6.7 7.3 7.9Sepal.Lengthupper region−boundregion weigths0.00.20.40.60.81.02 2.43 3.09 3.75 4.4Sepal.Widthupper region−boundregion weigths0.00.20.40.60.81.00.994 2.4 3.53 4.65 5.78 6.91Petal.Lengthupper region−boundregion weigths0.00.20.40.60.81.00.0976 0.766 1.3 1.7 2.1 2.5Petal.Widthupper region−boundregion weigths0.00.20.40.60.81.0Fig. 1. Example: Visualization of a result for Iris data4 Parameter choice for DiSCoFor the implementation one has to find the Characteristic Regions. So theproblem is how to form the regions and how to choose the thresholds.4.1 Building the regionsAs mentioned earlier the corresponding frequencies should be proportionalto heights of histogram bars for convenience so we can refer to the theory ofnonparametric density estimation to build the regions. In histogram densityestimation the problem consists in smoothing but not over-smoothing theempirical distribution of the data. Thus the bandwidth of a histogram shouldbe chosen neither too small nor too large. Freedman and Diaconis (1981)suggest a choice ofbw =23√NIQR (7)as bandwidth where IQR is the interquartile range. Under weak assumptionsthis histogram is L2-convergent for density estimation (Freedman and Dia-conis, 1981). As the distribution may be different in the classes this must6 Szepannek and Luebkebe done for every class – and every variable. The number of classwise binsis then Md(k) = ⌊xd(Nk)−xd(1k)bw(k,d) ⌋ with xd(Nk) and xd(1k) being the classwisemaximum respective minimum and ⌊.⌋ being the rounding operator. WithIV d := [xd(1), xd(N)] and IVdk := [xd(1k), xd(Nk)] let:Md :=⌊ ∫IV d1dt∫∪kIVdk1dt{∑k(Md(k)∫IV dk{∑kI[IV dk](s)}−1ds)}⌋(8)This means that the classwise number of bins is interpolated resp. averagedfor intervals covered by none, one or more than one class. So the regions ofvariable d are IV d divided into Md equal parts. Rd0 and RdMd+1 cover theupper and lower rest.4.2 Optimizing the thresholdsThere remains the question how to choose the thresholds in equation 2 andequation 3. So far no theoretical background is known for an optimal choiceof both SDR (Dense Regions) and SRR (Relevant Regions).The optimal parameters are found by a contracting 2-dimensional grid-search algorithm. As the criterion for optimization the cross validated errorrate is used. It should be noticed that since the number of observations is finitesmall changes of the two thresholds will not change the resulting model. Inorder to check the parameters one can consider that a rather small thresholdSDR eliminates outliers but keeps a large probability mass in the remainingregions. A SRR rather large keeps only regions in the model that stronglyindicate one class.5 Simulation study5.1 Data generationIn order to obtain more general results an experimental design is used in datageneration to be able to compare the effects of possibly influencing factorsin the data on the classification result of DiSCo and of two well-establishedother methods: Classification Trees (CART) and Linear Discriminant Anal-ysis (LDA).With the factor levels described below, data of 8 or 12 variables are firstdrawn from independent normal distributions with variance 1 but differentexpectations in 3 classes. These data are transformed to possess differentkurtosis and skewness and to be deflected.Below we give a brief description of the seven investigated factors:• The class priors may be equal or not.DiSCo - Different Subspace Classification 7• We investigated two different class mean settings in the first 6 variables:either only one class mean separated from the others or all three classmeans are different (one in the middle between the others). For 3 variablesthe doubled 0.95 quantile is chosen, for the 3 other variables the doubled0.9 quantile of the standard normal distribution is chosen for the tallestdifferences in the location of the class means.• 2 or 6 irrelevant independent variables are attached to the data thatare N(0, 1) distributed for all classes, i.e. either a quarter or half of thevariables do not contain any separating information.• All variables are transformed to have high or low kurtosis and skewnessfollowing the Johnson-System (see Johnson, 1949) to generate a widerange of values in the kurtosis-skewness-plane.• The probability of an object to be deflected is fixed to be 0.1 or 0.4, wheredeflection means that an object is moved into one of two directions: halfthe distance towards its class mean or half the way away from it into thedirection of its nearest wrong class.The factors and levels included in the experimental design of the simulationstudy are summarized in Table 1. A Plackett-Burman design (Plackett andBurman, 1946) for these factors was repeated 20 times.Effect Low level High levelClass priors ( 13, 13, 13) ( 23, 16, 16)Number of different class means 3 2Added irrelevant variables 2 6Kurtosis 2.7 5Skewness 0.12 1.152Probability to be deflected 0.1 0.4Direction of deflection towards class mean away from class meanTable 1. Effects and levels on the simulated data sets5.2 ResultsCompared are both proposed classification rules for the DiSCo method in-cluding (labelled (1)) and not including (2) the adjoining regions, CART(Breiman et al., 1984) and LDA. Table 2 shows the mean error rates on thetest data and the estimated effects of the main factors (coded to −1/ + 1)used in the design (cp. table 1) on log(odds(hitrate)). These effects can beestimated independently by a regression on the coded influencing factors:DiSCo seems to outperform the Classification trees and is almost as goodas LDA. One can also see that there are only small differences between bothproposed classification rules for the DiSCo method so there is no general rulewhich one to use.8 Szepannek and LuebkeDiSCo (1) DiSCo (2) CART LDAOverall mean error 0.085 0.079 0.127 0.075Class Priors 0.41 0.48 0.19 0.24Number of different class means 0.17 0.24 0.34 0.37Irrelevant variables 0.19 0.26 0.21 -0.11Kurtosis -0.12 -0.07 -0.27 0.09Skewness 1.16 1.06 0.89 0.70Probability to be deflected -0.10 -0.16 -0.24 -0.61Deflected direction -2.17 -2.23 -1.00 -2.73Table 2. Results: Overall mean error and estimated effects on log(odds(hitrate))It can be concluded that LDA has best overall mean error. Classificationtrees perform well with deflection away from the class mean but having alarge general deficit. The DiSCo method, having a good average result, ispreferable with skewed data or differing class priors and a high percentage ofdeflected objects.Mean values for the optimal thresholds are SDR = 0.67 and 0.54 includingand not including the neighbour regions while the averaged optimal SRR are1.88 and 1.75.6 SummaryThe introduced concept of Characteristic Regions allows the visualizationof the class characteristics and so satisfies the aim of an easy comprehensionand interpretation of the data. It also yields intuitive classification rules.On simulated test data it outperformed classification trees and was almostas good as the linear discriminant analysis.ReferencesBreiman, L., Friedman, J., Olshen, R. & Stone, C. (1984): Classification and re-gression trees. Chapman & Hall.Freedman, D. and Diaconis, P. (1981): On the histogram as a density estimator:L2 theory. Zeitschrift fu¨r Wahrscheinlichkeitsthoerie und verw. Gebiete, 57,453-476.Johnson, N. L. (1949): Bivariate distributions based on simple translation systems.Biometrika, 36, 149-176.Plackett, R., Burman, J. (1946): The design of optimum multifactorial experiments.Biometrika, 33, 305-325.

https://www.econstor.eu/dspace/bitstream/10419/22582/1/tr69-04.pdf

Different Subspace Classification

Abstract

Similar works

Full text

Available Versions

EconStor (ZBW Kiel)

Eldorado - Ressourcen aus und für Lehre, Studium und Forschung