Proceeding of: 7th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2003 Maó, Menorca, Spain, June 3–6, 2003.The design of nearest neighbour classifiers is very dependent from some crucial parameters involved in learning, like the number of prototypes to use, the initial localization of these prototypes, and a smoothing parameter. These parameters have to be found by a trial and error process or by some automatic methods. In this work, an evolutionary approach based on Nearest Neighbour Classifier (ENNC), is described. Main property of this algorithm is that it does not require any of the above mentioned parameters. The algorithm is based on the evolution of a set of prototypes that can execute several operators in order to increase their quality in a local sense, and emerging a high classification accuracy for the whole classifier

D. Aha

K. Z. Mao

P. Burrascano

English

Crossref

Evolutionary Approach to Overcome Initialization Parameters in Classification Problems

Isasi, Pedro

Fernández, Fernando

Name not available

Evolutionary approach to overcome initialization parameters in classification problems

Universidad Carlos III de Madrid e-Archivo

Evolutionary Approach to Overcome InitializationParameters in Classification ProblemsP. Isasi,F. FernandezComputer Science departmentUniversidad Carlos III de MadridAbstract The design of nearest neighbour classifiers is very dependent fromsome crucial parameters involved in learning, like the number of prototypes touse, the initial localization of these prototypes, and a smoothing parameter. Theseparameters have to be found by a trial and error process or by some automaticmethods. In this work, an evolutionary approach based on Nearest NeighbourClassifier (ENNC), is described. Main property of this algorithm is that it doesnot require any of the above mentioned parameters. The algorithm is based onthe evolution of a set of prototypes that can execute several operators in order toincrease their quality in a local sense, and emerging a high classification accuracyfor the whole classifier.1 INTRODUCTIONNearest Neighbour Classifiers are defined as the sort of classifiers that assign to eachnew unlabelled example, vr2 V = fv1; : : : ; vMg, the label of the nearest prototype,ri, from a set, C = fr1; : : : ; rNg, of N prototypes previously classified [DH73]. Theset of different labels or classes is S = fs1; : : : ; sLg.The design of these classifiers is difficult, and rely in the way of defining the numberof prototypes needed to achieve a good accuracy, as well as the initial set of prototypesused and, usually, a smoothing parameter. Many discussions about what is the righttechnique to use can be found in the literature. Some approaches based on clusteringtechniques are divided in two main steps. The first one is to cluster a set of unlabelledinput data to obtain a reduced set of prototypes, for instance, with the LBG algorithm.The second step is to classify these prototypes basing on previosly labelled examplesand the nearest neighbour rule. Although this approach produces good results, it is ob-vious that to introduce information about the classification performance in the locationof the prototypes seems to be needed to achieve a higher performance.Neural networks approaches are also very common in the literature, like the LVQalgorithm [Koh84] and the works with radial basis functions. Some techniques try to in-troduce or to eliminate prototypes (or neurons) while designing the classifier followingdifferent heuristics, as the average quantization distortion or the accuracy in the classi-fication. Other approaches try to define first the optimal size of the classifier and thento learn it using the previous value.In this work, an evolutionary approach called Evolutionary Nearest Neighbour Clas-sifier (ENNC), is introduced to dynamically define the number of prototypes of theclassifier as well as the location of these prototypes.                12 THE ENNC ALGORITHM2.1 ArchitectureThe system can be represented by a bidimensional matrix, where each row is associatedto a region ri2 C, and each column is associated to a class sj2 S. Each position (i; j)of the matrix is a structure that contains features about the set of training examplesthat belongs to the region riand to the class sj. These sets are called Region-ClassSets, and notated by Vij. Furthermore, information about the elements that belong to adefined class (set V Sj), and that belong to a defined region (set V Ri) are stored. Last,some information can be obtained from the set V . Note that the structure does not storethe whole region, class, and region-class sets, but only some features about them, astheir size, or their centroid, that will be defined below. For all the sets, its membershipfunction is given too.Figure 1 shows the architecture used to keep all this information. Each of thosenodes stores the features about the sets that represent. This structure is dynamic, in thesense that if any set is empty, the node is not created, solving some memory require-ments in complex domains.V VC1VRVCLVCjVRVR1Ni... ................................. ...... ............ ... ...V11 V VVVVVV V1j 1LiLNLNjijN1i1Region−Class SetsPattern Set Class SetsRegion SetsFigure1. ENNC ArchitectureFrom this architecture some important features are computed:– regionssjis the number of regions which prototype class is sj.– expectationsjis the number of patterns that any prototype riof the class sjisexpected to correctly classify. This number is computed in a relative way, by therelationship among the number of patterns and the number of regions of each classsj:expectationsj=kV Sjkregionssj(1)– locationriis the location of the prototype ri.– classriis the class of the prototype ri.– accuracyriis the classification accuracy of the prototype. It is calculated as shownin equation 2, being si= classri.accuracyri=kVisikkV Rik(2)         2– apportationriis a relationship among the number of patterns correctly classifiedby this prototype and the number of patterns that it is suppose to classify:apportationri=kVisikexpectationsi2(3)– qualityriis the quality of the prototype, calculated from equation 4. This valueis a relationship among the accuracy of the prototypes, and its contribution to theclassification of the whole input pattern set.qualityri= max(1; accuracyri apportationri) (4)– neighbours(ri) is the set of regions that has a common border with the region ri.2.2 The AlgorithmThe learning phase is an iterative process where the prototypes can execute several op-erators, once the features defined in section 2.1 have been computed. This operators areheuristics that allow the prototypes to change their location, to introduce new proto-types, etc. The algorithm begins with an initialization where the classifier is composedby only one prototype. After, the classifier evolves by executing, in a loop, all the de-signed operators, described below. These operators take advantage of some informationgathered at the beginning of the loop.Initialization One relevant feature of our method is the absolute elimination of initialconditions. These initials conditions are usually summarized in three: the number ofprototypes, the initial set of prototypes and a smoothing parameter. The ENNC algo-rithm allows to learn without those parameters, given that the initial number of proto-types is always one, the initial location of that prototype is not relevant, and there areno learning parameters.Mutation Operator The goal of this operator is to label each prototype with the mostpopulate class in each region. Once the features are obtained, each prototype knowsthe number of patterns of each class located in its region, so the prototype changes, ifneeded, and becomes to the same class of the most abundant class of patterns in itsregion.This way to get the main class is typically used when unsupervised learning is ap-plied to supervised classification. In these works, the algorithms typically generate a setof clusters that takes into account only the distribution of the data. In a second phase, theclusters are labelled to the most populate class in the cluster. However, in this work, thesupervision is included in each iteration of the algorithm, and not only in “a posteriori”phase. The formulation of this operator is defined in equation 5.8ri2 C; classri= argjmax kVijk (5)where kVijk is the size of the set Vij.    3Reproduction Operator The goal of this operator is to introduce new prototypes in theclassifier. The insertion of new prototypes is a decision that is taken by each prototype,in such a way that all of them contains in its region as much patterns as possible ofthe same class. The regions with patterns belonging to different classes, can create newregions containing the patterns of a different class to the class of the prototype.Each prototype, ri, of the class sj, executes a roulette proportional to the number ofelements of the set Vij0 to which it represents. The possible results of the roulette aretwo. On on hand, if the resulting set Vij0 is the set Vij, i.e. j = j 0, no reproduction isexecuted. On the other hand, if the resulting set Vij0 is not the set Vij, i.e. j 6= j 0, thereproduction is executed, and a new region ri0 is created to contain the patterns in Vij0 ,that is renamed to be Vi0j0 .Fight Operator This operator provides the prototype the capability of getting patternsfrom other regions. Formally, this operator allows a prototype, ri, to modify its setsVijfrom the sets Vi0jof another prototype ri0 , for i 6= i0. This operator takes place inseveral steps:1. Choose the prototype ri0 against to fight proportionally to the difference betweenits quality and the quality of a neighbor ri.2. Decide whether to fight or not proportionally to the distance of their qualities:Pfight(ri; ri0) = jqualityri  qualityri0j (6)3. If prototype ridecides to fight with prototype ri0 , there are two possibilities. Givenclassri= si, and classri0= si0 :– If si6= si0 (cooperation). Both prototypes belongs to different classes. In thiscase, the prototype ri0 will give to the prototype rithe patterns of the class si.– If si= si0 (competition). In this case, patterns can be transfered from set Vi0sito the set Visi, or vice versa, depending on who wins the fight. The amountof patterns that are transferred depends on a probability proportional to thequalities of both prototypes.Move Operator The move operation implies to relocate each prototype in the bestexpected place. So each prototype, ri, decides to move to the centroid of the set Vij,being classri= j, i.e. it takes as centroid the centroid of the set Vijof its class, asshown in equation 7.locationri= centroidVisi(7)where si= classri.This operation based in the second step of Lloyd iteration allows to make a localoptimization of the classifier, increasing its performance.Die Operator The probability for a prototype of being eliminated is 1 minus the doubleof the quality, as defined in equation 8. In that case, successful prototypes will survivewith probability of 1, while useless prototypes with quality less than 0.5 might die. A         4wide range of heuristics about how to reduce the number of prototypes in a classifiercan be found in the bibliography.Pdie(ri) =0; when qualityri> 0:51  2  qualityri; when qualityri 0:5(8)3 ExperimentsThis section shows the experiments performed over two different domains, where theENNC algorithm is compared with other classification algorithms, as C4.5 [Qui93],decission rules [FW98], Naive Bayes [DH73], and IBK [AK91], for values of k = 1and k = 3, executed in WEKA [WF00].3.1 Spiral Data SetSpiral data set is typically used in the bibliography as a challenge data set, where twointerlazed spirals must be correctly classified, as it is shown in figure 2. Examples in thesame spiral belong to the same class, and there are 500 examples in each spiral. Fromthe whole set, 900 examples was used for learning, and 100 for test.2000 −1500 −1000 −500 0 500 1000 1500 200class 1 class 2Figure2. Spiral Data SetGiven that the ENNC algorithm has an important stochastic component, the ex-periment has been executed 20 times, each of them of 300 iterations. The results aresummarized in Table 1 where, for each execution, the iteration when the winner classi-fier was evolved is shown, as well as the number of prototypes of the classifier and theclassification error for training and test sets. The table shows that the 99% of success istypically achieved on training, while success on test set ranges from 94% to 98%, whatmeans a maximum difference of 4 errors over the test set. The number of prototypes ofthe classifiers achieved in each iteration is close to 80.    5Iteration Prototypes Training (%) Test (%) Iteration Prototypes Training (%) Test (%)289 82 99,33 97,0 281 73 98,9 96,0158 75 99,0 96,0 300 84 99,44 97,0174 81 99,11 95,0 123 84 99,56 96,0267 81 99,33 96,0 81 79 99,33 94,0282 81 99,44 97,0 260 82 99,44 96,0260 78 99,11 97,0 131 77 98,67 95,0170 79 99,56 96,0 81 81 99,33 98,0182 82 99,33 97,0 97 79 99,33 96,0120 79 98,67 95,0 299 78 99,56 96,0291 84 99,78 97,0 179 75 99,0 97,0Table1. Results of different executions of the ENNC algorithm over spiral data set.Table 2 shows comparative results of different classifiers over this domain, showingthe best and the lowest results of the ENNC algorithm over the 20 executions. Thesame training and test sets was used with all the classifier systems. Results for C4.5,decission rules (PART) and Naive Bayes are poor, and they are far from the solutionsobtained by ENNC. However, IBK achieves the best results of a 100% of success as isexpected in a domain with the examples of different classes very separated, as shownin Figure 2.C4.5 PART Naive Bayes IBK (k = 1) IBK (k = 3) ENNC (best) ENNC (worst)62 56 50 100 100 98 94Table2. Comparative results over the spiral data set.Figure 3 shows the evolution of one execution of the ENNC algorithm, with thenumber of prototypes and success obtained in each iteration over the training and thetest sets. Figure shows that in only 25 iterations, the number of prototypes grows up to64, obtaining a success of the 91,34% over the training set, and of the 85% over thetest set. However, the number of prototypes still grows up to the range 70-80, whenthe search of better solutions go on, obtaining the best solution in iteration 176, with asuccess of 94% over the test set.3.2 Uniformly Distributed DataIn this experiment, the ENNC algorithm has been executed over the data set shown inthe Figure 4, as defined in [Bur91]. As in that work, data set contains 2000 instances ofeach class, using 500 for training ant 1500 for test.As in previous case, the ENNC algorithm is executed 20 times, with a mamimumlength of 300 iterations. The algorithm achieves a solution of only 2 prototypes in 16of the 20 executions, achieving a 98.6% iver the training and test sets. Other executionsachieve better results for classifiers with more prototypes, arriving up to the 98.73%with 12 prototypes. Table 3 shows some comparative results with previous approaches,as LVQ [Koh84] and LVQ-PNN [Bur91] of different sizes, and a different version of         601020304050607080901000 50 100 150 200 250 30Success and Number of PrototypesIterationsTraining Test PrototypesFigure3. Evolution of one execution of the ENNC algorithm over spiral data set.0 20 40 60 80 100class 1 class 2Figure4. Uniformly Distributed Data.PNN [MTS00] that automatically defines the number of neurons. Furthermore, C4.5,PART, Naive Bayes and IBK (experimented with the default values defined in WEKA)are compared too.The table shows that the worst result obtained by ENNC for two prototypes (98.6%)is very similar to the best solutions obtained with other different approaches, while thebest solution obtained, with a 98.93% of success, is better than the rest of solutionsreported.4 CONCLUSIONSThe algorithm presented in this work is an evolutionary approach to solve the problemof finding a set of prototypes that are able to correctly classify the examples of a domain,following a 1-nearest neighbour approach. The main advantage of this method are, on    7Algorithm Data SuccessLVQ 10 prototypes 96.13LVQ 100 prototypes 97.77LVQ-PNN 10 Neurons 96.99LVQ-PNN 100 Neurons 98.17PNN (Mao) 8 Neurons 98.85C4.5 - 96.97PART - 96.6Naive Bayes - 96.77IBK k = 1 98.7IBK k = 3 98.93ENNC (best) 2 Prototypes 98.93ENNC (worst) 2 Prototypes 98.6Table3. Comparative results over Uniformly Distributed Data.one hand, that it is able to achieve a high accuracy in most of the domains where ithas been tested, even compared with other techniques from the literature. On the otherhand, the achievement of these good results is done without the user defines the initialconditions for learning.References[AK91] D. Aha and K. Kibler. Instance-based learning algorithms. Machine Learning, 6:37–66,1991.[Bur91] Pietro Burrascano. Learning vector quantization for the probabilistic neural network.IEEE Transactions on Neural Networks, 2(4):458–461, July 1991.[DH73] Richard O. Duda and Peter E. Hart. Pattern Classification and Scene Analysis. JohnWiley And Sons, 1973.[FW98] Eibe Frank and Ian H. Witten. Generating accurate rule sets without global optimiza-tion. In Proceedings of the Fifteenth International Conference on Machine Learnin,1998.[Koh84] Teuvo Kohonen. Self-Organization and Associative Memory. Springer, Berlin, Heidel-berg, 1984. 3rd ed. 1989.[MTS00] K. Z. Mao, K.-C. Tan, and W. Ser. Probabilistic neural-network structure determinationfor pattern classification. IEEE Transactions on Neural Networks, 11(4):1009–1016,July 2000.[Qui93] J. Ross Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, 1993.[WF00] I. H. Witten and E. Frank. Data Mining. Practical Machine Learning Tools and Tech-niques with Java Implementations. Morgan Kaufmann, 2000.         8

Isasi Viñuela, Pedro

http://e-archivo.uc3m.es/bitstream/handle/10016/3991/evolutionary_LNCS_2003_ps.pdf?sequence=1

Evolutionary approach to overcome initialization parameters in classification problems

Abstract

Similar works

Full text

Available Versions

Crossref

Name not available

Universidad Carlos III de Madrid e-Archivo

Name not available