Archambeau, C.

Butz, T.

Popovici, V.

Thiran, J.

Verleysen, M.

Infoscience - École polytechnique fédérale de Lausanne

Supervised Nonparametric Information Theoretic ClassificationCe´dric Archambeau†, Torsten Butz‡, Vlad Popovici‡, Michel Verleysen∗,†, Jean-Philippe Thiran‡† Universite´ catholique de LouvainMachine Learning GroupPlace du Levant, 31348 Louvain-la-Neuve, Belgium{archambeau,verleysen}@dice.ucl.ac.be‡ Swiss Federal Institute of Technology (EPFL)Signal Processing Institute1015 Lausanne, Switzerland{torsten.butz,vlad.popovici,jp.thiran}@epfl.chAbstractIn this paper, supervised nonparametric informationtheoretic classification (ITC) is introduced. Its principlerelies on the likelihood of a data sample of transmittingits class label to data points in its vicinity. ITC’s learn-ing rule is linked to the concept of information potentialand the approach is validated on Ripley’s data set. Weshow that ITC may outperform classical classification al-gorithms, such as probabilistic neural networks and sup-port vector machines.1. IntroductionClustering and classification algorithms are fundamen-tal ingredients in pattern recognition, data mining, know-ledge discovery and other related fields [4, 1]. Whereasclustering is an unsupervised way of grouping data usinga measure of similarity, classification can be seen as its su-pervised version. Indeed, when the labels of the class pro-totypes are available, one should use this additional know-ledge in order to construct or improve a classification rule.Subsequently, this classification rule may be used to auto-matically classify new data samples. This rule is thereforeoften called the generalization rule and the set of labelledprototypes can be seen as the learning data set.A common technique for clustering and classificationinvolves the estimation of the underlying probability den-sity functions (PDFs) of the classes. Two types of ap-∗M.V. is a Senior Research Associate of the Belgian National Fundfor Scientific Research.This work was partially supported by the European Commission (IST-2000-25145), the Belgian FNRS and the Swiss NCCR IM2, funded bythe Swiss National Science Foundation.proaches can be considered. In parametric PDF estima-tion, it is assumed that the data is drawn from a specificdensity model. The model parameters are then fitted to thedata. Unfortunately, an a priori choice of the PDF modelis not always suited in practice, as it might provide a falserepresentation of the true PDF. By contrast, we can buildnonparametric PDF estimators, as for example the Parzenwindow estimator [6]. Such techniques do not assume anyfunctional form of the PDF and allow its shape to be en-tirely determined from the data.Recently, a clustering evaluation function based on theInformation Potential was proposed [5] in order to per-form nonparametric clustering tasks. In [2] the formalismof information theoretic clustering is generalized by con-sidering nonparametric classification error estimation andby linking it to information theoretical concepts, such aserror transmission and distortion.Within this framework, we propose a classification cri-terion strongly related to information theory, as it allowsto minimize the error transmission. Actually, assigningthe optimal class label to a data sample can be seen asminimizing the erroneous transmission of the real classlabel of that sample. In addition, the proposed classifica-tion algorithm is nonparametric. As a consequence, theshape of the true, but unknown underlying PDFs is notenforced a priori.This paper is organized as follows. In Section 2, we in-troduce nonparametric information theoretic classification(ITC), which is based on the local class label transmission.Subsequently, in Section 3, we recall the K-nearest neigh-bor exchange algorithm [4], which is used for optimizingthe learning rule. Finally, in Section 4, we show simu-lation results using Ripley’s synthetic data set [8]. Wecompare the performance of ITC to probabilistic neuralnetworks [9], which are nothing else than Bayesian clas-0-7695-2128-2/04 $20.00 (C) 2004 IEEEsification combined to nonparametric class PDF estima-tion, and support vector machines [10, 3], using Gaussiankernels. All the considered methods are supervised.2. Information theoretic classificationIn this section, the concepts of local and global classlabel transmission are first defined. Subsequently, an in-formation theoretic learning rule related to error transmis-sion and the information potential is constructed. Finally,a classification criterion based on local class label trans-mission is introduced.2.1. Local class label transmissionConsider the d-dimensional feature vector x ∈ Ωx. Letus define the following likelihood function:L(x|C,C ′) = p(x|C) · p(x|C ′), (1)where C ∈ ΩC and C ′ ∈ ΩC are the true class labeland the estimated class label of x respectively. The quan-tity L(x|C,C ′) corresponds to the likelihood of havingthe data sample x given C and C ′. In other words, it isthe likelihood of having x knowing that its true class labelis C and supposing we assign it the label C ′. Therefore(1) characterizes the local transmission of the label C tothe estimated label C ′ in the vicinity of x.2.2. Global class label transmissionWe may extend the concept of local label transmissionby defining the global transmission of the class labels asthe expected likelihood of transmitting the true class labelC to the estimated label C ′ over the entire feature domainΩx:T (C,C ′) =∫Ωx∑C∈ΩC∑C′∈ΩC D(C,C′)· L(x|C,C ′)P (C)P (C ′)dx. (2)In this equation, P (C) and P (C ′) are the true and esti-mated class priors, respectively. The function D(C,C ′)is the similarity function.In this paper, we focus on the minimization of the errortransmission, i.e. we want to maximize the global trans-mission of true class labels. Besides, we do not assumeany class-dependent similarity for simplicity. Therefore,we assume D(C,C ′) equals 1 when C = C ′ and 0 other-wise. For other similarity choices, we refer to [2]. Themain advantage of the similarity function is that it in-creases the flexibility of the classification algorithm bymaking it application-dependent.2.3. Classification learning ruleThe Parzen window estimator [6] is one of the mostpopular techniques for estimating a PDF nonparametri-cally. It consists of placing a well-defined kernel functionon each data sample and then determining a common ker-nel width σ. In practice, Gaussian kernels are often used.The estimated PDF is defined as the sum of all Gaussiankernels, multiplied by a normalization factor.Both conditional PDFs appearing in the expression ofthe likelihood (1), can be estimated nonparametricallyfrom the data by the Parzen window estimator:pˆ(x|C) = 1|SC |∑xi∈SCN(x− xi, σi2), (3)pˆ(x|C ′) = 1|SC′ |∑xj∈SC′N(x− xj , σj2), (4)where the Gaussian kernels are defined as follows:N(x−m, σ2) = 1(2πσ2)d/2exp[−‖x−m‖22σ2]. (5)The sets SC and SC′ contain the data samples with classlabels C and C ′ respectively, and |SC | and |SC′ | are theircardinality.Posing M the number of prototypes, substituting thePDF estimates (3) and (4) into expression (2) of the globalclass label transmission T (C,C ′), and replacing the priorsP (C) and P (C ′) by |SC |M and|SC′ |M respectively, leads to:T (C,C ′) =∫Ωx∑C∈ΩC∑C′∈ΩC D(C,C′)· 1M∑xi∈SC N(x− xi, σi2)· 1M∑xj∈SC′ N(x− xj , σj2)dx.After integration, we can write the following informa-tion theoretic learning rule:Cˆ ′ = argmaxC′∑C∈ΩC∑C′∈ΩCD(C,C ′)V (SC , SC′), (6)whereV (SC , SC′) =1M2∑xi∈SC∑xj∈SC′N(xi − xj , σi2 + σj2).The quantity V (SC , SC′) is the information potential [5]and is closely related to Renyi’s entropy [7].Note that in classical nonparametric PDF estimation,the kernel width enforces a smooth density model. How-ever, within the information theoretic framework, it canbe interpreted in a slightly different way. Here, the kernelwidth controls the extent of the local domain around eachlabelled prototype, rather than the degree of overlappingof the kernels. In other words, it regulates the region oflabel transmission of each data point.0-7695-2128-2/04 $20.00 (C) 2004 IEEE2.4. Classification criterionBy solving the maximization problem posed by the in-formation theoretic learning rule, we can automaticallyclassify the class prototypes. Yet, whenever one wantsto classify new data samples the whole optimization pro-cedure of the classification algorithm should be repeated,which is not suited in practice. In addition, when the ker-nel widths are not chosen properly, we might easily overfitthe data, resulting in a classification scheme not represen-tative of the true classes. Therefore, class prototypes areused for learning the classification task, while test sam-ples are kept for evaluating its performance. The optimalkernel widths can then be selected according to the bestgeneralization performance. It remains that, in order totest the algorithm, we need a valid classification rule.Consider again the information theoretic learning rule.By applying this rule, we maximize the global transmis-sion of the correct class labels over the entire feature spacedomain. As a result, we assign the most probable classlabels Cˆ ′l to the class prototypes {xl}Mll=1, according totheir neighborhood. By analogy to (2) where C = Cˆ ′land C ′ = Cˆ ′l , we can classify the test samples {xt}Mt=11according to the likelihood of their local estimated classlabel transmission, weighted by the class priors:Cˆ ′t = argmaxcLˆ(xt|c = Cˆ ′l , c = Cˆ ′l)P (Cˆ ′l)P (Cˆ ′l). (7)Although the classification learning rule and the classifi-cation criterion are not identical, it was found experimen-tally that both are in agreement when the kernel widthsare not overestimated.3.K-nearest neighbor exchange algorithmThe learning rule (6) can be optimized by the K-nearest neighbor exchange algorithm [4]. This optimiza-tion procedure allows escaping from local minima andsaves computations by labelling data groups instead of theindividual data points. As the contribution to the globallabel transmission of data samples lying closely to eachother and belonging to the same classes is large, the clas-sification algorithm assigns a common class label to sam-ples in the same neighborhood. Meanwhile, class bound-aries are positioned along regions of low sample density.The optimization algorithm iterates over the followingsteps:1. Initialize randomly a class label to each sample of thelearning data set {xl}Mll=1 and choose an initial groupsize K = K0;−1.5 −1 −0.5 0 0.5 1 1.5−0.200.20.40.60.811.2Figure 1. Ripley’s synthetic learning dataset. The true class of the prototypes aredenoted by ‘◦’ and ‘+’ respectively.2. Create Ml groups by searching for each sample xlthe K-nearest neighbors with the same class label;3. Remove identical groups, resulting in Pl groups;4. Repeat until no improvement:• For each group p, change its class label to anyclass label and record improvement if any;• If in the previous step, any improvement wasrecorded, permute randomly the group indices;5. If K > 1, divide the group size K by two, and go tostep 2;6. End of algorithm.In practice, the initial group size K0 can be initialized forexample as Ml|ΩC | .4. Results and discussionThe proposed supervised information theoretic classifica-tion algorithm is validated on Ripley’s data set [8], shownin Figure 1. It is a synthetic data set, composed of a learn-ing set of size Ml = 250 and a test set of size Mt = 1000.Each class is a mixture of two Gaussian distributions. Thetwo equally sized classes are strongly overlapping.In this paper, the performances of information theoreticclassification (ITC), probabilistic neural networks (PNNs)and support vector machines (SVMs) are compared. Weconsider PNNs using Gaussian activation functions. Ba-sically, such PNNs perform Bayesian classification afterestimating the PDF of each class by means of the classprototypes [9]. The class densities are estimated non-parametrically by Parzen. SVMs are supervised classi-fication technique based on Vapnik’s statistical learning0-7695-2128-2/04 $20.00 (C) 2004 IEEE−1.5 −1 −0.5 0 0.5 1 1.5−0.200.20.40.60.811.2PNN (sigma=0.125)−1.5 −1 −0.5 0 0.5 1 1.5−0.200.20.40.60.811.2SVM (sigma=0.77, c=15)−1.5 −1 −0.5 0 0.5 1 1.5−0.200.20.40.60.811.2ITC (sigma=0.2)Figure 2. Classification of Ripley’s synthetic test set. The left figure shows the classification byPNN, the one in the middle by SVM and the figure on the right by ITC.theory [10, 3]. The goal is to find a large margin separa-tion hyper-plane between the two classes. As the classesare usually not linearly separable in the input space, datais projected into a high-dimensional space through the so-called kernel-trick, where the linear classification can bedone. Again, we consider Gaussian kernels.In Figure 2, the classification result of Ripley’s test setis shown, using PNN, SVM and ITC. In the latter, wehave taken σi = σj = σITC for simplicity. By per-forming an exhaustive search, it was found that the opti-mal kernel width for PNN is σPNN = 0.125 and for ITCσITC = 0.2. As noted in Section 2.3, this difference canbe explained by the information theoretical interpretationgiven to the kernel width in the context of ITC, i.e. it con-trols the region of label transmission around each proto-type. The optimal kernel width for SVM is σSVM = 0.75and the regularization factor c = 15. For each classifi-cation algorithm, the average classification error E wascomputed. For PNN we have obtained EPNN = 9.3%,for SVM ESVM = 9.7%, and for ITC EITC = 9.1%.PNN and ITC perform better than SVM. When com-paring PNN and ITC, one can see that ITC performsslightly better than PNN. This can be explained as follows.The effect of learning the classification task by maximiz-ing the global class label transmission is the removal ofclass outliers when estimating the class PDFs. We denoteby the term ‘class outlier’ a data sample located within aforeign class. Hence, the class transitions are steeper, re-ducing the uncertainty on the location of the class bound-aries. The classification result is therefore less sensitiveto atypical data samples, as they do not contribute to thePDF estimation of their respective classes.5. ConclusionIn this paper, supervised nonparametric informationtheoretic classification was introduced. The resulting clas-sification algorithm exploits the likelihood of each classprototype of transmitting its class label to a data pointlocated in its vicinity. Based on this principle, an infor-mation theoretic learning rule and a related classificationrule were proposed. The former is linked to the con-cept of information potential. The relevance of the ap-proach was demonstrated by simulation. Indeed, infor-mation theoretic classification outperforms probabilisticneural networks and support vector machines on Ripley’swell-known synthetic data set.References[1] C. M. Bishop. Neural Networks for Pattern Recognition.Oxford university press, 1995.[2] T. Butz. From Error Probability to Information Theo-retic Signal and Image Processing. Ph.D. dissertation,Swiss Federal Institute of Technology (EPFL), Switzer-land, 2003.[3] N. Cristianini and J. Shawe-Taylor. An Introduction toSupport Vector Machines and other Kernel-based Learn-ing Methods. Cambridge university press, 2000.[4] R. O. Duda and P. E. Hart. Pattern Classification andScene Analysis. Wiley, 1973.[5] E. Gokcay and J. C. Principe. Information theoretic clus-tering. IEEE Patt. Anal. Mach. Int., 24(2):158–171, 2002.[6] E. Parzen. On estimation of a probability density functionand mode. Ann. Math. Stat., 33:1065–1076, 1962.[7] A. Renyi. On measures of entropy and information. InProc. of 4th Berkeley Symp. on Prob. Math. Stat., vol-ume 1, pages 547–561, 1960.[8] B. Ripley. Pattern Recognition and Neural Networks.Cambridge University Press, 1996.[9] D. F. Specht. Probabilistic neural networks. Neural Net-works, 3:109–118, 1990.[10] B. Vapnik. The Nature of Statistical Learning Theory.Springer, 1995.0-7695-2128-2/04 $20.00 (C) 2004 IEEE

Supervised nonparametric information theoretic classification

http://infoscience.epfl.ch/record/87125/files/Archambeau2004_905.pdf

Supervised nonparametric information theoretic classification

Abstract

Similar works

Full text

Available Versions

Infoscience - École polytechnique fédérale de Lausanne