There is a great need for classification methods that can properly handle asymmetric cost and benefit constraints of classifications. In this study, we aim to emphasize the importance of classification benefits by means of a new classification algorithm, Benefit-Maximizing classifier with Feature Intervals (BMFI) that uses feature projection based knowledge representation. Empirical results show that BMFI has promising performance compared to recent cost-sensitive algorithms in terms of the benefit gained

Güvenir H.A.

Ikizler, N.

Bilkent University Institutional Repository

Maximizing Benefit of ClassificationsUsing Feature IntervalsNazlı İkizler and H. Altay GüvenirBilkent University Department of Computer Engineering06533, Ankara Turkey{inazli,guvenir}@cs.bilkent.edu.trAbstract. There is a great need for classification methods that canproperly handle asymmetric cost and benefit constraints of classifica-tions. In this study, we aim to emphasize the importance of classificationbenefits by means of a new classification algorithm, Benefit-Maximizingclassifier with Feature Intervals (BMFI) that uses feature projectionbased knowledge representation. Empirical results show that BMFI haspromising performance compared to recent cost-sensitive algorithms interms of the benefit gained.1 IntroductionClassical machine learning applications try to reduce the quantity of the errorsand usually ignore the quality of errors. However, in real-world applications, thenature of the error is very crucial. Further, the benefit of correct classificationsmay not be the same for all cases. Cost-sensitive classification research addressesthis imperfection and evaluates the effects of predictions rather than simplymeasuring the predictive accuracy. By incorporating cost(and benefit) knowledgeto the process of classification, the effectiveness of the algorithms in real-worldsituations can be evaluated more rationally. In this study, we concentrate oncosts of misclassifications and try to minimize those costs, by maximizing thetotal benefit gained during the process of classification.Within this framework, we propose a new cost-sensitive classification tech-nique, called Benefit-Maximizing classifier with Feature Intervals (BMFI forshort),that uses the predictive power of feature projection method previouslyproposed in [6]. In BMFI, voting procedure has been changed to impose thecost-sensitivity property. Generalization techniques are implemented to avoidoverfitting and to eliminate redundancy. BMFI has been tested over severalbenchmark datasets and a number of real-world datasets that we have compiled.The rest of the paper is organized as follows: In Section 2, benefit maximiza-tion problem is addressed. Section 3 gives the algorithmic descriptions of BMFIalgorithm along with the details of feature intervals concept, voting method andgeneralizations. Experimental evaluation of BMFI is presented in Section 4. Fi-nally, Section 5 reviews the results and presents future research directions onthe subject.V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 339–345, 2003.c© Springer-Verlag Berlin Heidelberg 2003340 Nazlı İkizler and H. Altay Güvenir2 Benefit Maximization ProblemRecent research in machine learning has used the terminology of costs when deal-ing with misclassifications. However, those studies mostly lack the informationthat correct classifications may have different interpretations. Besides implyingno cost, accurate labeling of instances may entail indisputable gains. Elkan pointsout the importance of these gains [3]. He states that doing accounting in termsof benefits is commonly preferable because there is a natural baseline from whichall benefits can be measured, and thus, it is much easier to avoid mistakes.Benefit concept is more appropriate to real world situations, since net flowof gain is more accurately denoted by benefits attained. If a prediction is prof-itable from the decision agent’s point of view, its benefit is said to be positive.Otherwise, it is negative, which is the same as cost of wrong decision. To incor-porate this natural knowledge of benefits to cost-sensitive learning, we have usedbenefit matrices. B=[bij] is a n × m benefit matrix of domain D if n equals tothe number of prediction labels, m equals to the number of possible class labelsin D and bij ’s are such thatbij ={≥ 0 if i = j< bii otherwise. (1)Here, bij represents the benefit of classifying an instance of true class j as class i.The structure of the benefit matrix is similar to that of the cost matrix, with theextension that entries can either have positive or negative values. In addition,diagonal elements should be non-negative values, ensuring that correct classifi-cations can never have negative benefits. Given a benefit matrix B, the optimalprediction for an instance x is the class i that maximizes expected benefit (EB),that isEB(x, i) =∑jP (j|x) × bij . (2)where P (j|x) is the probability that x has true class j. The total expected benefitof the classifier model M over the whole test data isEBM =∑xarg maxi∈CEB(x, i) =∑xarg maxi∈C∑jP (j|x) × bij . (3)where C is the set of possible class labels in the domain.3 Benefit Maximization with Feature IntervalsAs shown in [6], feature projections based classification is a fast and accuratemethod, and the rules it learns are easy for humans to verify. For this reason,we have chosen to extend its predictive power to involve benefit knowledge.In a particular classification problem, given the training dataset which con-sists of p features, an instance x can be thought as a point in a p-dimensionalspace with an associated class label xc. It is represented as a vector of nominal orMaximizing Benefit of Classifications Using Feature Intervals 341train(TrainingSet, BenefitMatrix)beginfor each feature fsort(f , TrainingSet)i list← make point intervals(f ,TrainingSet)for each interval i in i listvotei(c)← voting method (i,f ,BenefitMatrix)if f is lineari list← generalize(i list,BenefitMatrix)end.Fig. 1. Training phase of BMFIlinear feature values together with its associated class, i.e., <x1, x2, .., xp, xc>.Here, xf represents the value of the fth feature of the instance x. If we considereach feature separately, and take x’s projection onto each feature dimension,then we can represent x by the combination of its feature projections.Training process of BMFI algorithm is given in Fig. 1. In the beginning,for each feature f , all training instances are sorted with respect to their valuefor f . This sort operation is identical to forming projections of the traininginstances for each feature f . A point interval is constructed for each projection.Initially, lower and upper bounds of the interval are equal to the f value of thecorresponding training instance. If the f value of a training instance is unknown,it is simply ignored. If there are several point intervals with the same f value,they are combined into a single point interval by adding the class counts. Atthe end of point interval construction, vote for each class label is determined byusing one of the two voting methods. The first one is the voting method of CFIalgorithm [5], called VM1 in our context. VM1 can be formulated as follows:V M1(c, I) =NcclassCount(c). (4)where Nc is the number of instances that belong to class c in interval I andclassCount(c) is the total number of instances of class c in the entire trainingset. This voting method favors the prediction of minority class in proportion toits occurrence in the interval. The second voting method, called VM2, is basicallyfounded on optimal prediction approximation given by (2) and makes direct useof the benefit matrix. VM2 casts votes to class c in interval I asV M2(c, I) =∑k∈Cbck × P (k|I) . (5)P (k|I) is the estimated probability that an instance falling to interval I willhave the true class k, and is calculated asP (k|I) = NkclassCount(k). (6)342 Nazlı İkizler and H. Altay Güvenirgeneralize(interval list)beginI ← first interval in interval listwhile I is not empty doI ′ ← interval after II”← interval after I ′if merge condition(I, I ′, I”) is truemerge I ′(and/or) I” into Ielse I ← I ′end.Fig. 2. Generalization of intervals step in BMFIAfter the initial assignment of votes, for linear features, intervals are general-ized to form range intervals in order to eliminate redundancy and avoid overfit-ting. The generalization process is illustrated in Fig. 2. Here, merge condition() isa comparison function that evaluates relative properties of each interval and re-turns true if sufficient level of similarity between neighboring intervals is reached.Besides adding more prediction power to the algorithm, proper generaliza-tion reduces the number of intervals, and by this way, decreases the classificationtime. In this work, we have experimented with three interval generalization meth-ods. The first one, called SF (same frequent) joins two consecutive intervals ifthe most frequently occurring class of both are the same. The second method,SB (same beneficial) joins two consecutive intervals if they have the same ben-eficial class. A class c is the beneficial class of an interval i iff for ∀j ∈ C andj = c, ∑x∈i B(x, c) ≥ ∑x∈i B(x, j) . If the beneficial classes of two consecutiveintervals are the same, then it can be more profitable to unite them into a singleinterval. The third method, HC (high confidence) combines three consecutive in-tervals into a single one, when the middle interval has less confidence on its votesthan the other two. The confidence of an interval is measured as the differencebetween votes of the most beneficial class and second beneficial class.Table 1. List of evaluated cost-sensitive algorithmsName DescriptionMetaNB MetaCost on Naive BayesMetaJ48 MetaCost on J4.8C1NB CostSensitiveClassifier with reweighting on Naive BayesC2NB CostSensitiveClassifier with direct minimization on Naive BayesC1J48 CostSensitiveClassifier with reweighting on J4.8C2J48 CostSensitiveClassifier with direct minimization on J4.8C1VFI CostSensitiveClassifier with reweighting on VFIC2VFI CostSensitiveClassifier with direct minimization on VFIMaximizing Benefit of Classifications Using Feature Intervals 343classify(q)beginfor each class cvc ← 0for each feature fif qf is knownI ← search interval(f, qf )for each class cvc ← vc + interval vote(c, I)prediction← argmaxc(vc)end.Fig. 3. Classification step in BMFIThe classification process of the BMFI algorithm is given in Fig. 3. The choiceof voting method to be used depends on the characteristics of the domain. Basedon our empirical results, we propose to use VM1 voting together with SF, SBand HC techniques when the correct classification of the minority class is morebeneficial than the other classes. On the contrary, when the benefit matrix isnot correlated with the distribution, VM2 can be employed together with SBand HC to boost up the benefit performance. Experimental results presented inSect. 4 are achieved by using this general rule-of-thumb.4 Experimental ResultsFor evaluation purposes, we have used benchmark datasets from UCI ML Repos-itory [1]. These data sets do not have predefined benefit matrices, so we formedtheir benefit matrices in the following manner. In binary datasets, one class isassumed to be more important to predict correctly than the other by a con-stant benefit ratio, b. We have tested our algorithm by using five different bvalues that are 2, 5, 10, 20, 50. Note that when b is equal to 1, the problemreduces to the classical classification problem. Further, we have compiled fournew datasets. Their benefit matrices have been defined by experts of each do-main. For more information about the datasets and benefit matrices the readeris referred to [7]. We have compared BMFI with MetaCost [2] and CostSensitive-Classifier of Weka [4] on well-known base classifiers which are Naive BayesianClassifier, C4.5 decision tree learner and VFI [6]. Table 1 lists these algorithmswith their base classifiers (J4.8 is Weka’s implementation of C4.5 in Java).MetaCost is a wrapper algorithm that takes a base classifier and makes itsensitive to costs of classification [2]. It operates with a bagging logic beneathand learns multiple classifiers on multiple bootstrap replicates of the trainingset. MetaCost has become a benchmark for comparing cost-sensitive algorithms.In addition to MetaCost, we have compared our algorithm with two cost sensi-tive classifiers provided in Weka. The first method uses reweighting of training344 Nazlı İkizler and H. Altay GüvenirTable 2. Comparative evaluation of BMFI with wrapper cost-sensitive algo-rithms. The entries are benefit per instance values. Best results are shown inbolddomain MetaNB MetaJ48 C1NB C2NB C1J48 C2J48 C1VFI C2VFI BMFIbreast-cancer 4.0 3.8 4.0 4.0 3.9 3.7 3.7 2.8 3.9 (VM1)pima-diabetes 2.8 2.8 3.0 2.7 2.9 2.5 -1.5 2.8 2.7 (VM1)ionosphere 5.7 6.5 6.1 6.0 6.5 5.7 6.4 6.1 6.5 (VM2)liver disorders 5.3 5.3 5.2 5.4 5.4 4.4 4.3 5.3 5.4 (VM2)sonar 3.3 4.6 4.5 4.0 4.6 3.3 0.0 4.0 4.9 (VM2)bank-loans -0.8 -0.4 -0.9 -0.6 0.1 -0.5 -1.2 -2.8 -0.1 (VM1)bankruptcy 7.8 7.5 7.7 7.4 7.5 7.3 7.7 7.8 7.9 (VM1)dermatology 7.5 7.2 7.5 7.5 7.2 7.3 6.9 5.6 7.4 (VM2)lesion 8.7 7.8 8.9 9.0 7.8 7.7 6.4 4.0 9.0 (VM1)instances and the second method makes direct cost-minimization based on prob-ability distributions [8]. We call these two classifiers C1 and C2, respectively.Experimental results are presented in Table 2. In this table, results of binarydatasets are benefit per instance values for b=10. All results are recorded byusing 10-fold cross validation. As the results demonstrate, BMFI algorithm isvery successful in most of the domains and remarkably comparable to other al-gorithms in all of the domains. In ionosphere, liver, sonar, bankruptcy and lesiondomains, BMFI attains the maximum benefit per instance value. In the remain-ing datasets its performance is very high and comparable to other algorithms.We have observed that benefit achieved is highly dependent on the nature of thedomain, i.e., benefit matrix information, distribution of classes, etc, as expected.In addition, it is worthwhile to note that BMFI outperforms cost-sensitiveversions of its base classifier VFI (C1VFI and C2VFI). This observation suggeststhat using benefit knowledge inside the algorithm itself is more effective thanwrapping a meta-stage around to transform it into a cost-sensitive classifier.In binary datasets, we observed that the success of BMFI increases as thebenefit ratio increases. This is an important highlight of BMFI and is mostlydue to its high sensitivity to benefit of classifications. This aspect of BMFI hasbeen illustrated with the results of pima-diabetes dataset given in Table 3.Table 3. Benefit per instance values of pima-diabetes dataset with differentbenefit ratios. Best results are shown in boldb MetaNB MetaJ48 C1NB C2NB C1J48 C2J48 C1VFI C2VFI BMFI2 0.5 0.6 0.7 0.6 0.6 0.6 0.0 0.0 0.55 1.2 1.2 1.5 1.3 1.2 1.2 -0.5 1.1 1.210 2.8 2.8 3.0 2.7 2.9 2.5 -1.5 2.8 2.720 5.8 5.8 6.2 6.1 6.1 5.6 -3.3 6.3 6.350 16.6 16.2 16.2 16.6 16.3 14.7 -9.0 16.7 16.8Maximizing Benefit of Classifications Using Feature Intervals 3455 Conclusions and Future WorkIn this study, we have focused on the problem of making predictions when theoutcomes have different benefits associated with them. We have implementeda new algorithm, namely BMFI that uses the predictive power of feature intervalsconcept in maximizing the total benefit of classifications. We make direct use ofbenefit matrix information provided to the algorithm in tuning the predictionso that the resultant benefit gain is maximized.BMFI has been compared to MetaCost and two other cost-sensitive classifi-cation algorithms provided in Weka. These generic algorithms are wrapped overNBC, C4.5 and VFI. The results show that BMFI is very effective in maximizingthe benefit per instance values. It is more successful in domains where the pre-diction of a certain class is particularly important. Empirical results we obtainedalso show that using benefit information directly in the algorithm itself is moreeffective than using a meta-stage around the base classifier.In benefit maximization problem, we have observed that individual character-istics of the datasets influence results significantly, due to the extreme correlationbetween cost-sensitivity and class distributions.As future work, feature-dependent domains can be explored in depth andfeature-dependency aspect of BMFI can be improved. Benefit maximization canbe extended to include the feature costs. Feature selection mechanisms that aresensitive to individual costs of features can be utilized.References[1] Blake C. L. and Merz C. J.: UCI repository of machine learning databases. Uni-versity of California, Irvine, Department of Information and Computer Sciences(1998) [http://www.ics.uci.edu/˜mlearn/MLRepository.html] 343[2] Domingos P.: Metacost: A general method for making classifiers cost-sensitive. In:Proceedings of the International Conference on Knowledge Discovery and DataMining, San Diego, CA (1999) 155-64 343[3] Elkan C.: The Foundations of Cost-Sensitive Learning. In: Proceedings of theSeventeenth International Joint Conference on Articial Intelligence (2001) 340[4] Frank E. et al.: Weka 3 - Data Mining with Open Source Machine LearningSoftware in Java. The University of Waikato (2000)[http://www.cs.waikato.ac.nz/˜ml/weka] 343[5] Güvenir H. A.: Detection of abnormal ECG recordings using feature intervals. In:Proceedings of the Tenth Turkish Symposium on Artificial Intelligence and NeuralNetworks (2001) 265-274 341[6] Güvenir H. A, Demiröz G., and İlter N.: Learning differential diagnosis oferythemato-squamous diseases using voting feature intervals. Artificial Intelli-gence in Medicine, Vol. 13(3) (1998) 147-165 339, 340, 343[7] İkizler N.: Benefit Maximizing Classification Using Feature Intervals. TechnicalReport BU-CE-0208, Bilkent University (2002) 343[8] Ting K. M.: An instance weighting method to induce cost-sensitive trees. IEEETransactions on Knowledge and Data Engineering, Vol. 14(3) (2002) 659-665 344

Maximizing benefit of classifications using feature intervals

http://repository.bilkent.edu.tr/bitstream/11693/27510/1/Maximizing_benefit_of_classifications_using_feature_intervals.pdf

Maximizing benefit of classifications using feature intervals

Abstract

Similar works

Full text

Available Versions

Bilkent University Institutional Repository