The class imbalance problem has been considered a critical factor for designing and constructing the supervised classifiers. In the case of artificial neural networks, this complexity negatively affects the generalization process on under-represented classes. However, it has also been observed that the decrease in the performance attainable of standard learners is not directly caused by the class imbalance, but is also related with other difficulties, such as overlapping. In this work, a new empirical study for handling class overlap and class imbalance on multi-class problem is described. In order to solve this problem, we propose the joint use of editing techniques and a modified MSE cost function for MLP. This analysis was made on a remote sensing data . The experimental results demonstrate the consistency and validity of the combined strategy here proposedPartially supported by the Spanish Ministry of Education and Science under grants CSD2007–00018, TIN2009–14205–C04–04, and by Fundació Caixa Castelló–Bancaixa under grants P1–1B2009–04 and P1–1B2009–45; SDMAIA-010 of the TESJO and 2933/2010 from the UAE

Alejo Eleuterio, Roberto

García Jiménez, Vicente

Martínez Sotoca, José

Valdovinos Rosas, Rosa María

García, Vicente

Repositori Institucional de la Universitat Jaume I

Back Propagation with Balanced MSE Cost Functionand Nearest Neighbor Editing for Handling ClassOverlap and Class Imbalance?R. Alejo1, J.M. Sotoca2 and V. Garcı´a2, and R.M. Valdovinos31 Tecnolo´gico de Estudios Superiores de Jocotitla´nCarretera Toluca-Atlacomulco KM. 44.8, Col. Ejido de San Juan y San Agustn, 50700 Jocotitla´n(Mexico)2 Institute of New Imaging Technologies, Universitat Jaume IAv. Sos Baynat s/n, 12071 Castello´ de la Plana (Spain)3 Centro Universitario UAEM Valle de Chalco, Universidad Auto´noma del Estado de Me´xicoHermenegildo Galena No.3, Col. Ma. Isabel, 56615 Valle de Chalco (Mexico)Abstract. The class imbalance problem has been considered a critical factorfor designing and constructing the supervised classifiers. In the case of artificialneural networks, this complexity negatively affects the generalization process onunder-represented classes. However, it has also been observed that the decrease inthe performance attainable of standard learners is not directly caused by the classimbalance, but is also related with other difficulties, such as overlapping. In thiswork, a new empirical study for handling class overlap and class imbalance onmulti-class problem is described. In order to solve this problem, we propose thejoint use of editing techniques and a modified MSE cost function for MLP. Thisanalysis was made on a remote sensing data . The experimental results demon-strate the consistency and validity of the combined strategy here proposed.Keywords: Multi-class imbalance, Overlapping, backpropagation, cost function,editing techniques.1 IntroductionClass imbalance constitutes one of the problems that has recently received most atten-tion in research areas such as Machine Learning, Pattern Recognition and Data Mining.The class imbalance occurs when some classes heavily outhumber other classes. Inthe area of the artificial neural networks (NN) has been observed that the class imbal-ance problem causes important losses in the generalization capacity when the minorityclasses [1, 2] are learned, because these are often biased towards the majority class. This? Partially supported by the Spanish Ministry of Education and Science under grants CSD2007–00018, TIN2009–14205–C04–04, and by Fundacio Caixa Castello–Bancaixa under grantsP1–1B2009–04 and P1–1B2009–45, FE28/2009 (103.5/09/4195) of the Mexican PROMEP,2703/2008U from the UAEM project and SEP-2003-C02-44225 from the Mexican CONA-CyT.issue can be found in real–world applications from Government, Industry and Academicor Scientific Area [3–6].Research on this topic can be roughly classified into three categories: assigning dif-ferent classification error costs [7], resampling the original training set, either by over-sampling the minority class and/or under-sampling the majority class until the classesare approximately equally represented [8, 9], and internally biasing the discrimination-based process so as to compensate for the class imbalance [10, 11].Recently, several works have pointed out that there does not exist a direct correlationbetween class imbalance and the loss of performance. These studies suggest that theclass imbalance is not a problem by itself, but the degradation of performance is alsorelated to other factors, such as the degree of overlapping between classes [12–14].In this paper, we propose to combine two strategies for addressing the class overlapand the class imbalance for the classification of remote sensing data. The problem is ofgreat relevance since very few approaches to deal with this challenge. In order to facesuch a problem, this work focus on the joint use of editing techniques and a modificationin the mean square error (MSE) cost function for a multi–layer Percetron (MLP). Thisapproach can be considered a two–stage method. Firstly, we remove noisy and border-line samples of the majority classes by application of editing techniques. Secondly, theedited data set is used for training a MLP with a modified MSE cost function, whichovercomes the class imbalance problem.2 Methodology2.1 A Balanced MSE Cost Function for Backpropagation AlgorithmIn the multilayer perceptron neural network (MLP) the training by Backpropagationalgorithm is based on minimization of a cost function. One of the most popular usedcost functions is the mean-square error (MSE) between the desired dzi and the actualyzi outputs for each class i = 1, 2 . . . J ,Ei(U) =1Nni∑z=1(dzi − yzi)2 , (1)where N =∑Ji ni is the total training samples and ni is the size of class i.For a two-class problem (J = 2) the mean square error function can be expressedas,E(U) =J∑i=1Ei = E1(U) + E2(U) . (2)If n1 << n2 then E1(U) << E2(U) and ‖∇E1(U)‖ << ‖∇E2(U)‖, conse-quently∇E(U) ≈ ∇E2(U), which means that −∇E(U).To obtain a balanced MSE cost function, we introduce a parameter (γ) that balancethe contributions of the MSE,E(U) =∑Ji=1 γ(i)Ei = γ(1)E1(U) + γ(2)E2(U) (3)where γ(1)‖∇E1(U)‖ ≈ γ(2)‖∇E2(U)‖ avoiding that the minority class be ignoredin the learning process. In this work, the parameter γ is defined asγ(i) = ‖∇Emax(U)‖/‖∇Ei(U)‖, (4)where ‖∇Emax(U)‖ corresponds to the largest majority class. When γ is includedin the training process, the data probability distribution is altered [11]. However, thisparameter (Eq. 4) reduces the impact in the data distribution probability because thecost function value is diminished gradually. In this way, the class imbalance problemis reduced in early iterations, and later γ(J) reduces its effect on the data distributionprobability.2.2 Editing techniquesThe editing techniques have been proposed to remove noise prototypes and possibleoverlap among classes from the training set. The aim is improve the classifier accuracyby producing smooth decision boundaries. One the most popular editing schemes isbased on the well-know k-NN rule, which is mainly used for classification. However,this rule only take into account the distances to a number of close neighbors. Alternativeconcepts of neighborhood have been proposed to consider the neighbors of a sample interms of proximity and spatial distribution (Surrounding Neighborhood).The editing techniques was used to remove noisy samples of the majority classesbut keeping all the positive examples. This task allows to improve the learning mechan-ics of the MLP. In next paragraphs we describe briefly basic concepts about editingalgorithms.Wilson Editing Wilson [15] developed the Edited Nearest Neighbor (ENN) algorithmin which S starts out the same as Traning Set (TS), and then each instance in S is re-moved if it does not agree with the majority of its k nearest neighbors (with k=3, typi-cally). This eliminates noisy instances as well as close border cases producing smootherdecision boundaries. Algorithmically, the ENN scheme can be expressed as follows:1. Let S = X .2. For each xi in X do:– Discard xi from S if it is misclassified using the k-NN rule with prototypes inX− {xi}.Editing Via Surrounding Approaches The Nearest Centroid Neighborhood (NCN)[16] refers to a concept in which neighborhood is defined taking into account the prox-imity of prototypes to a given input sample and maintaining their symmetrical distri-bution around it. The k-Nearest Centroid Neighborhood rule(k-NCN) [17] has beenproved to overcome the traditional k-NN classifier in many practical situations. TheNCN Editing (NCNE) approach corresponds to slight modification of the original workof Wilson and basically consists of using the error estimated by the k-NCN classifica-tion rule.Proximity graph editing scheme is based on the concepts of Gabriel Graph (GG)and Relative Neighborhood Graph (RNG) [18]. The method applies Wilson’s editingalgorithm [15] using proximity graphs (GG or RNG) for each sample instead of theEuclidean distance.The Gabriel Graph Editing (GGE) and Relative Neighborhood Graph Editing (RNGE)can be summarized as follows: after computing the graph neighborhood of every samplein the original training set, discard those samples that are misclassified by their graphneighbors (instead of their k nearest neighbors).These editing techniques provide some advantages in comparation to conventionalmethods. GGE and RNGE get some kind of information about prototypes close enoughbut homogeneously distributed around a given sample, which can be specially interest-ing to detect outliers close to the inter-class or decision boundaries. A more detaileddescription of GGE and RNGE can be found in [19].2.3 Random under-samplingRandom under-sampling aims at balancing the data set through the random removal ofnegative examples. Despite its simplicity, it has empirically been shown to be one ofthe most effective resampling methods.In this work, the random under-sampling is used to compare with the editing tech-niques, also, it was not hired to balance the training set.3 Experimental Set–UpIn this part, a comparative was carried out among the strategies previously describedto validate the methodology exposed in Section 2. The database used corresponds toremote-sensing task which is basically a multi-classification problem.Experiments were conducted as follows:Data set: A large data set, the Cayo data (4 bands, 11 classes) which corresponds to aspectral image with reference to a particular region in the Gulf of Mexico was em-ployed in the experiments. The data set was transformed into five-class problems(MCayo) by joining the samples of several classes. The fourth column in Table 1indicates the original classes that have been joined to shape the new classes. Forinstance, the samples of clases 1, 3, 6, 7, and 10 were combined to form the classC–01 and the original classes 2, 4 and 5 were left as C–02, C–04 and C–05, respec-tively.Partitions: A stratified 10–fold cross–validation was employed.Under–sampling strategies: random under–sampling (RUS), nearest centroid neigh-borhood editing (NCNE), Wilson’s editing (ENN), relative neighborhood graphediting (RNGE) and Grabiel graph editing (GGE) were employed. All these tech-niques were applied over the majoriy classes. In the case of ENN and NCNE, thevalue of k has been set to 15 and 13, respectively.Classifiers: We use a MLP with and without balanced cost function (Cost-MLP). Eachone was trained with the back–propagation algorithm in batch mode. The followingparameter settings were used: a learning rate η = 0.1 and one hidden layer withseven neurons.Performance metrics: Overall accuracy, accuracy by class and the geometric meanof accuracies measured seperately on each class were used. These measures can beeasily derived from am×mmatrix confusion as that given in Table 2. Thus overallaccuracy is computed as Accuracy =∑mi=1 nii/N , where N is the total numberof samples, Accuracy by class = nii/ni+ and the geometric mean as g-mean =(∏mi=1 nii/ni+)1m .Table 1. Number of training and testing samples in each classNew Classes Training Test Original Classes %C-01 2689 299 1,3,6,7,10 49.64C-02 264 29 2 4.87C-03 2055 228 8,9,11 37.93C-04 290 32 4 5.35C-05 120 13 5 2.21Both MLP as Cost-MLP classifiers were trained using each original and prepro-cessed training data set by different editing techniques.Table 2. Confusion matrix for a multi-class problemReal ClassesPredicted Classes 1 2 · · · m total (ni+)1 n11 n12 · · · n1m n1+2 n21 n22 · · · n2m n2+...............m nm1 nm2 · · · nmm nm+total (n+j) n+1 n+2 · · · n+m N4 Results and discussionSeveral experiments with MCayo database were developed in the experimental process.’Cost-MLP’ denotes the balanced MSE cost function with MLP and ’TS edited’ is thetraining set edited.Table 3 shows the overall accuracy and the g-mean obtained with the approachespreviously described. We can observe that the classification accuracy is high and the g-mean is low. So, the minority samples are missclassified while the samples of majorityclasses are well identified. When the original data set is classified with Cost-MLP bothperformance measures are improved.On other hand, when the editing techniques are employed, the g-mean is improvedthan the original training data set (without preprocessing). The classification resultsobtained from the joint application of editing techniques and Cost-MLP outperform theg-mean with respect to apply the two techniques separately.We observe that the RUS algorithm, although shows a slightly improvement, theediting techniques appears as the best strategies. Analyzing the percentage of reduction,higher values are obtained for the editing techniques that obtain better g-mean.Table 3. Experimental results by editing the majority classesMLP Original ENN NCNE RNGE GGE RUSAccuracy 83.27(1.20) 85.50(1.43) 84.65(2.00) 84.96(1.29) 85.55(1.34) 84.95(1.46)g-mean 00.00(0.00) 43.18(27.27) 65.19(12.15) 22.85(26.84) 47.06(25.94) 37.43(26.01)Cost-MLP Original ENN NCNE RNGE GGE RUSAccuracy 86.40(1.06) 83.37(2.32) 82.60(2.57) 84.64(1.84) 83.59(2.27) 86.25(1.21)g-mean 69.80(3.14) 81.14(4.26) 82.26(3.73) 77.10(5.51) 82.05(4.42) 71.97(3.90)% reduction 00.00(0.00) 25.00(0.46) 33.00(0.55) 19.00(0.19) 28.00(0.33) 33.00(0.08)In Table 4 we can see the results of editing techniques and Cost-MLP for each class.The two first columns indicate the strategy applied and the number class. The thirdcolumn we show the proportion of class elements in relation with the total samples(ratio = ni/N , where ni is the elements number of class i and N the total samplesin the TS). The fourth column is the classification accuracy and the last one showsthe classes with the level of confusion which is greater than 10% (the percentage ofconfusion appears in brackets).In table 4, we can observe that the editing techniques reduce the confusion levelamong classes. For example, Class 2 is overlapped with the class 1. When it is appliedthe editing techniques, the confusion level is diminished. In this case, the NCNE obtainsa better performance.5 ConclusionIn this paper, we analyze how to deal the class overlap and class imbalance in a multi–classifcation problem. The goal was study the performance of these two techniquescombined: editing techniques joint balanced MSE cost function with MLP.The experiments show that the benefits associated to inclusion a balanced MSE costfunction in the training process. However, this is not enough for reducing the overlap-ping among classes. For that, using the edition strategies, we can reduce the overlappingproblem increasing the prediction of the minority classes. In this paper, the use of bothTable 4. Performance on each class with the Cost-MLPClass Ratio Accuracy % confusion ( > 10 %)C-01 0.49 88.32C-02 0.05 51.67 C-01 (48.26)Original C-03 0.38 93.69C-04 0.06 61.76 C-01 (34.25)C-05 0.02 63.91 C-01 (34.14)C-01 0.49 76.50C-02 0.05 88.50 C-01 (11.16)ENN C-03 0.38 93.37C-04 0.06 74.61 C-01 (15.91)C-05 0.02 76.62 C-01 (21.73)C-01 0.49 74.87C-02 0.05 91.40NCNE C-03 0.38 92.53C-04 0.06 76.91 C-01 (15.85)C-05 0.02 79.25 C-01 (19.17)C-01 0.49 81.53C-02 0.05 75.39 C-01 (24.37)RNGE C-03 0.38 93.23C-04 0.06 67.86 C-01 (24.61)C-05 0.02 72.93 C-01 (25.26)C-01 0.49 76.59C-02 0.05 91.60GGE C-03 0.38 93.34C-04 0.06 74.88 C-01 (17.53)C-05 0.02 77.59 C-01 (20.30)C-01 0.49 87.35C-02 0.05 55.02 C-01 (44.98)RUS C-03 0.38 93.57C-04 0.06 62.68 C-01 (33.39)C-05 0.02 70.08 C-01 (28.27)techniques is the best option to reduce the classification error and solve these kind ofproblems.Future works will be addressed to investigate the potential of these editing methodsapplied in the hidden space of the neural network. This involves working in the spaceof the hidden layer and not in the feature space, such as commonly happens with theWilson’s editing and its variants.References1. Anand, R., Mehrotra, K., Mohan, C., Ranka, S.: An improved algorithm for neural networkclassification of imbalanced training sets. IEEE Transactions on Neural Networks 4 (1993)962–9692. Ou, G., Murphey, Y.L.: Multi-class pattern classification using neural networks. PatternRecognition 40(1) (2007) 4–183. Al-Haddad, L., Morris, C.W., Boddy, L.: Training radial basis function neural networks:effects of training set size and imbalanced training sets. Journal of Microbiological Methods43(1) (2000) 33 – 444. He, H., Garcia, E.: Learning from imbalanced data. IEEE Transactions on Knowledge andData Engineering In Knowledge and Data Engineering 21(9) (2009) 1263–12845. Huang, Y.M., Hung, C.M., Jiau, H.C.: Evaluation of neural networks and data mining meth-ods on a credit assessment task for class imbalance problem. Nonlinear Analysis: Real WorldApplications 7(4) (2006) 720–7476. Mazurowski, M.A., Habas, P.A., Zurada, J.M., Lo, J.Y., Baker, J.A., Tourassi, G.D.: Trainingneural network classifiers for medical decision making: The effects of imbalanced datasetson classification performance. Neural Networks 21(2-3) (2008) 427–4367. Zhou, Z.H., Liu, X.Y.: Training cost-sensitive neural networks with methods addressingthe class imbalance problem. IEEE Transactions on Knowledge and Data Engineering. 18(2006) 63–778. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research 16 (2002) 321–3579. Garcı´a, S., Herrera, F.: Evolutionary undersampling for classification with imbalanceddatasets: Proposals and taxonomy. Evolutionary Computation 17 (2009) 275–30610. Anand, R., Mehrotra, K., Mohan, C., Ranka, S.: Efficient classification for multiclass prob-lems using modular neural networks. Neural Networks, IEEE Transactions on 6(1) (1995)117–12411. Bruzzone, L., Serpico, S.: Classification of imbalanced remote-sensing data by neural net-works. Pattern Recognition Letters 18 (1997) 1323–132812. Garcı´a, V., Mollineda, R.A., Sa´nchez, J.S.: On the k-nn performance in a challenging sce-nario of imbalance and overlapping. Pattern Analysis and Applications 11(3) (September2008) 269–28013. Prati, R., Batista, G., Monard, M.: Class imbalances versus class overlapping: An analysisof a learning system behavior. In: MICAI. (2004) 312–32114. Visa, S., Ralescu, A.: Learning imbalanced and overlapping classes using fuzzy sets. In:Workshop on Learning from Imbalanced Datasets(ICML0´3). (2003) 91–10415. Wilson, D.: Asymptotic properties of nearest neighbor rules using edited data. IEEE Trans-actions on Systems, Man and Cybernetics 2(4) (1972) 408–42016. Chaudhuri, B.B.: A new definition of neighborhood of a point in multi-dimensional space.Pattern Recognition Letters 17(1) (1996) 11–1717. Sa´nchez, J.S., Pla, F., Ferri, F.J.: On the use of neighbourhood-based non-parametric classi-fiers. Pattern Recognition Letters 18(11-13) (1997) 1179–118618. Jaromczyk, J., Toussaint, G.: Relative neighborhood graphs and their relatives. Proceedingsof the IEEE 80(9) (1992) 1502–151719. Sa´nchez, J.S., Pla, F., Ferri, F.J.: Prototype selection for the nearest neighbour rule throughproximity graphs. Pattern Recognition Letters 18(6) (1997) 507–513

Back propagation with balanced MSE cost Function and nearest neighbor editing for handling class overlap and class imbalance

Abstract

Similar works

Full text

Available Versions

Repositori Institucional de la Universitat Jaume I

Repositori UJI