De Baets, Bernard

Waegeman, Willem

English

Ghent University Academic Bibliography

In several application domains such as biol-ogy, computer vision, social network analy-sis and information retrieval, multi-class clas-sification problems arise in which data in-stances not simply belong to one particularclass, but exhibit a certain membership toall the classes. Often referred to as learn-ing fuzzy, mixed or partial memberships, thistype of problems has been recently studied infields like fuzzy set theory, statistics and ma-chine learning, mainly from an unsupervisedlearning perspective. Given the interpreta-tion of fuzzy class memberships in real-worldapplications, we present a supervised proba-bilistic approach. To this end, we show inparticular how kernel-based logistic regressionmodels can learn fuzzy memberships in anadequate manner, just by replacing zero-one-coded class labels by fuzzy labels in the like-lihood function. Empirical results on severalreal-world data sets show that our approachleads to quasi-identical results but a tremen-dous gain in computational complexity, whencompared with a naive algorithm that trans-forms each fuzzy class label into many zero-one coded class labels.Keywords: fuzzy/mixed/partial member-ship models, logistic regression, kernel meth-ods, machine learning.1 IntroductionAs a general introduction to the concept offuzzy membership models, let us start witha slightly controversial yet suitable exampleabout the classification of humans into eth-nic groups. Until the 18th century it hasbeen claimed by scientists that all humans canbe subdivided into five main classes, namelywhite, black, yellow, red and brown people.Nowadays such superseded views have beenrecognized as too restrictive: in which classdo we put for example a man with a Euro-pean mother and an Asian father? In somesense one can say that such a man obtainsa membership to both classes white and yel-low. So, hypothetically speaking, if we wouldlike to construct a machine learning algorithmfor classifying humans, for example based onDNA data, then we would definitely need aclassifier capable of handling fuzzy class la-bels.Actually, it turns out that many real-worldmulti-class classification problems can betranslated into a setting where non-crisp classlabels are observed. Scene classification incomputer vision is such an application, as im-ages can simultaneously belong to let’s saythe classes “sunsets” and “historic buildings”[Woods et al., 1995, Boutell et al., 2004]. Herea strong connection with multi-label classifi-cation and multi-label ranking can be seen[Fu¨rnkranz et al., 2008, Hu¨llermeier et al.,2008]. Similar arguments hold for relatedapplication domains such as text and scien-tific literature categorization [Erosheva et al.,2004] and social network analysis [Koutsoure-lakis and Eliassi-Rad, 2008].Other applications can be found in the life sci-ences. Fuzzy membership models have beenapplied to the classification of satellite im-ages for crop-land suitability analysis [Nisar-Ahamad et al., 2000]. In microbiology as well,fatty-acid profiles describe bacterial species interms of fuzzy memberships [Marttinen et al.,2008], and in biochemistry fuzzy membershipsare observed when chemical compounds areproduced in certain proportions, thereby de-pending on the chosen environmental condi-tions. Finally, biological experts in agricul-ture often screen a fixed number of plants toconstruct disease prediction models, such thatfor each field a fraction of these plants is clas-sified in each of the disease classes, again re-sulting in fuzzy class memberships [Isebaertet al., 2009].In the latter applications rather a link withprobabilistic classifiers and multi-class prob-ability estimation can be claimed. Indeed,from a probabilistic perspective, one can in-terpret fuzzy memberships as probability es-timates for the class labels or as class pro-portions, so that in essence for each data ob-ject in the training data set a multinomialdistribution over the class labels is observed.In this light fuzzy membership models andtraditional probabilistic classifiers both pro-duce probability estimates as outputs, butthey differ in the sense that the former mod-els directly take fuzzy memberships as train-ing labels, while the latter ones only acceptcrisp training labels as input. In some imple-mentations of generalized linear models, fuzzymemberships can be indirectly processed astraining labels for binary classification, butthis trick is at least not widely establishedand rarely supported by statistical packages[Agresti, 2002].Given the large number of potential applica-tions in various domains, researchers in statis-tics, machine learning and fuzzy sets haveshown interest in developing learning algo-rithms for fuzzy memberships. In statisticsand machine learning the notions mixed mem-bership or partial membership are mainlyestablished, and here previous research wasmainly focussed on unsupervised settings,such that in clustering algorithms a data in-stance can simultaneously exhibit a member-ship in several clusters. Such ideas have forexample been incorporated in mixed mod-els [Gormley and Murphy, 2008], probabilis-tic graphical models [Airoldi et al., 2008] andBayesian clustering techniques [Heller et al.,2008].In fuzzy set theory, unsupervised approacheshave been popular too, consider for exam-ple the well-known fuzzy c-means clusteringmethod, but here a few supervised algorithmhave been proposed as well [Cai et al., 2007,Orsenigo and Vercellis, 2007]. Nevertheless,these algorithms tend to depart from a sim-ilar setup as probabilistic classifiers, in thesense that they also assume crisp class labelsfor training data, but fuzzy memberships arepredicted instead of probability estimates.While assuming a probabilistic interpretationof fuzzy class memberships, we will in this ar-ticle discuss how the kernel logistic regressionmethod can be generalized so that fuzzy classlabels are accepted as input. We give in thefollowing section a brief summary of the basicalgorithm, followed by its extension for fuzzymemberships in Section 3.2 Kernel logistic regression formulti-class problemsWe start with introducing some notations.Let us in general assume a K-class classifica-tion problem in which the classes are formallydenoted as C1, ..., CK . For ease of notation,training data will consist of pairs of the form(x,y) in which x represents a feature vectorand y is a vector of length K such that the k-th entry is 1 when the corresponding instancehas label Ck, while all other entries are zero.This condition will be relaxed in the followingsection such that the vector y will representfuzzy memberships to all classes. Further-more, let us assume that examples are iden-tically and independently drawn according toan unknown joint distribution over an inputspace X and a label space Y. We define a dataset of size N as D = {(x1,y1), ..., (xN ,yN )}.We will use the notation yn = {yn1, ..., ynK}so that ynk denotes the k-th entry in the labelvector that corresponds with the n-th datainstance.Given this notation, multi-class logistic re-gression (KLR) methods estimate the prob-ability that a given data instance belongs toclass Ck as follows:pk(xn) = Pr{ynk = 1 | xn}=exp(fk(xn))∑Kl=1 exp(fl(xn)),in which f1, ..., fK : X → R are scoring func-tions that assign a continuous value to datainstances. In traditional logistic regressionmodels these scoring functions are just linearmodels. In kernel logistic regression models[Zhu and Hastie, 2004], they can be generallyrepresented in the following way:fk(x) = wk · φ(x) + b ,with φ representing a feature mapping to apossibly high-dimensional feature space andw1, ...,wK vectors of parameters that mustbe estimated based on training data. Ac-cording to the representer theorem [Scho¨lkopfand Smola, 2002], the scoring functions canbe equivalently expressed as follows:fk(x) =N∑n=1αnkK(xn,x) ,with K a kernel corresponding to φ and αnkdual parameters.In probabilistic models an estimate of theparameters is usually obtained by maximumlikelihood estimation. The multinomial likeli-hood is given by:L(w1, ...,wK) = Pr{y1, ...,yN | w1, ...,wK}=N∏n=1K∏k=1(pk(xn))ynk .Equivalently, we can minimize the negativelog-likelihood:lnL =N∑n=1K∑k=1ynk ln(pk(xn))=N∑n=1K∑k=1ynk(fk(x)− ln ( K∑l=1exp(fl(xn)))).In the two-class case, the minimum is usuallyfound with gradient descent. For the multi-class case, one arrives at a constrained opti-mization problem, since it must hold that:K∑k=1pk(x) = 1 , ∀x .Variants of the sequential minimal optimiza-tion algorithm found in implementations ofsupport vector machines have been proposedfor the multi-class case [Keerthi et al., 2005,Zhu and Hastie, 2005].3 Learning fuzzy membershipswith KLRIn order to extend kernel logistic regression sothat data objects can have fuzzy membershipsto all of the classes, we will allow now that y ∈[0, 1]K instead of y ∈ {0, 1}K . In addition,we will impose the following constraint on thelabel vectors:K∑k=1ynk = 1 , ∀n .This constraint results in a probabilistic in-terpretation of fuzzy class memberships, andit also guarantees that we rather stay in amulti-class classification setting instead of amulti-label classification setting. A naive andcomputationally inefficient approach for mod-elling fuzzy memberships with existing sta-tistical tools could consist of artificially cre-ating a new (much larger) dataset D∗ ={(x∗1,y∗1), ..., (x∗N∗ ,y∗N∗)} by multiplying theoriginal dataset size with a certain factor. Inthis new dataset we then assign crisp labelsclass to the duplicates of the original data in-stances, in accordance with the distributiongenerated by the fuzzy memberships, so thaty∗m ∈ {0, 1}K for all (x∗m,y∗m) ∈ D∗. For-mally, let the multiplication factor be τ , then|D∗| = τ × |D| andyn ' 1τN∗∑m=1:xm=xnym , n ∈ {1, ..., N} ,Basically, the larger τ is, the better the ap-proximation, but also the more intractablethis approach becomes, because the size of D∗blows up very fast. To overcome this compu-tational bottleneck, we can write down thelog-likelihood for D∗ as follows:lnL =N∗∑m=1K∑k=1y∗mk ln(pk(x∗m))=N∑n=1τ∑t=1K∑k=1y∗u(n,t)k ln(pk(xn))'N∑n=1K∑k=1ynk ln(pk(xn))=N∑n=1K∑k=1ynk(fk(xn)− ln ( K∑l=1exp(fl(xn)))).in which an index function u : N[1, N ] ×N[1, τ ] → N[1, N∗] is used in order to mapan object from D to its t-th duplicate in D∗.Thus, we can avoid the construction of D∗,while still a very similar log-likelihood func-tion is minimized. Beside computational ad-vantages, this approach is also conceptuallypreferred since no approximation is requiredany more.4 ExperimentsFigure 1: Example of figure title5 DiscussionIn this paper we considered multi-class clas-sification problems where uncertainty is ob-served in the class labels, so that data in-stances obtain a fuzzy membership to sev-eral classes. As an extension of kernel logis-tic regression methods, we presented a simple,yet effective supervised learning approach tomodel this type of data. To this end, it wasshown that the naive idea of multiplying theoriginal dataset and replacing the fuzzy la-bels by several crisp labels can be avoided,since fuzzy class labels can be included in thelog-likelihood in an elegant way. Initial ex-perimental results on synthetic and real-worlddata confirm that including the fuzzy mem-berships in the log-likelihood function leadsto quasi-identical results as multiplying thedataset. Simultaneously, the computationalburden of the latter approach can be avoided.To this end, an existing R implementation ofkernel logistic regression was modified. Dur-ing the talk we will explain more thoroughlythe empirical results.We would like to thank Ji Zhu from the Uni-versity of Michigan for providing R code onkernel logistic regression.ReferencesA. Agresti. Categorical Data Analysis, 2ndversion. John Wiley and Sons, 2002.E. Airoldi, D. Blei, S. Fienberg, and E. Xing.Mixed membership stochastic blockmodels.Journal of Machine Learning Research, 9:1981–2014, 2008.M. Boutell, J. Luo, X. Shen, and C. Brown.Learning multi-label scene classification.Pattern Recognition, 37:1751–1771, 2004.W. Cai, S. Chen, and D. Zhang. Robust fuzzyrelational classifier incorporating the softclass labels. Pattern Recognition Letters,28:2250–2263, 2007.E. Erosheva, S. Fienberg, and J. Laf-ferty. Mixed-membership models of scien-tific publications. PNAS, 101:5220–5227,2004.J. Fu¨rnkranz, E. Hu¨llermeier, E. Mencia, andK. Brinker. Multilabel classification by cal-ibrated label ranking. Machine Learning,73:133–153, 2008.I. Gormley and T. Murphy. A mixture ofexperts models for rank data with applica-tions in election studies. Annals of AppliedStatistics, 2:1452–1477, 2008.K. Heller, S. Williamson, and Z. Ghahramani.Statistical models for partial membership.In Proceedings of the 25th Conference onMachine Learning, Helsinki, Finland, pages392–399, 2008.E. Hu¨llermeier, J. Fu¨rnkranz, W. Cheng, andK. Brinker. Label ranking by learningpairwise preferences. Artificial Intelligence,172:1897–1916, 2008.S. Isebaert, S. De Saeger, R. Devreese,R. Verhoeven, P. Maene, B. Heremans,and G. Haesaert. Mycotoxin-producingfusarium species occurring in winter wheatin Belgium (Flanders) during 2002-2005.Journal of Phytopathology, 157:108–116,2009.S. Keerthi, K. Duan, S. Shevade, and A. Poo.A fast dual algorithm for kernel logistic re-gression. Machine Learning, 61:151–165,2005.P.S. Koutsourelakis and T. Eliassi-Rad. Find-ing mixed-memberships in social networks.In Proceedings of the AAAI Spring Sym-posium on Social Information Processing,Stanford, CA, USA, 2008.P. Marttinen, J. Tang, B. De Baets,P. Dawyndt, and J. Corander. Bayesianclustering of fuzzy feature vectors using aquasi-likelihood approach. IEEE Transac-tions on Pattern Analysis and Machine In-telligence, 31:74–85, 2008.T. Nisar-Ahamad, K. Gropal-Rao, andJ. Murthy. GIS-based fuzzy membershipmodel for crop-land suitability analysis.Agricultural Systems, 63:75–95, 2000.C. Orsenigo and C. Vercellis. Evaluatingmembership functions for fuzzy discreteSVM. Lecture Notes in Computer Science,4578:187–194, 2007.B. Scho¨lkopf and A. Smola. Learning withKernels, Support Vector Machines, Regu-larisation, Optimization and Beyond. TheMIT Press, 2002.K. Woods, D. Cook, L. Hall, K. Bowyer, andL. Stark. Learning membership functions ina function-based object recognition system.Journal of Artificial Intelligence Research,3:187–222, 1995.J. Zhu and T. Hastie. Classification of genemicroarrays by penalized logistic regres-sion. Biostatistics, 5:427–443, 2004.J. Zhu and T. Hastie. Kernel logistic regres-sion and the import vector machine. Jour-nal of Computational and Graphical Statis-tics, 14:185–205, 2005.

Modelling partial class memberships in multi-class classification: a probabilistic approach

https://biblio.ugent.be/publication/860684/file/860685

Modelling partial class memberships in multi-class classification: a probabilistic approach

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography