As an analysis of the classification accuracy bound for the Nearest Neighbor technique, in this work we have studied if it is possible to find a good value of the parmeter k for each example according to their attribute values. Or at least, if there is a pattern for the parameter k in the original search space. We have carried out different approaches based onthe Nearest Neighbor technique and calculated the prediction accuracy for a group of databases from the UCI repository. Based on the experimental results of our study, we can state that, in general, it is not possible to know a priori a specific value of k to correctly classify an unseen example

D. R. Wilson

D. W. Aha

D. Wettschereck

I. Tomek

M. Stone

R. C. Holte

S. A. Dudani

S. Cost

T. M. Cover

Crossref

Empirical Evaluation of the Difficulty of Finding a Good Value of k for the Nearest Neighbor

Ferrer Troyano, Francisco Javier

Aguilar Ruiz, Jesús Salvador

Riquelme Santos, José Cristóbal

idUS. Depósito de Investigación Universidad de Sevilla

Francisco J .  Ferrer-Troyano, Jes& S. Aguilar-Ruiz, and Jose C. Riquelme Department of Computer Science, University of Sevilla Avenida Reina Mercedes s/n, 4101 2 Sevilla, Spain (ferrer,aguilar,riquelme~@lsi.us.es Abstract. As an analysis of the classification accuracy bound for the Nearest Neighbor technique, in this work we have studied if it is possible to find a good value of the parameter k for each example according to their attribute values. Or at least, if there is a pattern for the parameter k in the original search space. We have carried out different approaches based on the Nearest Neighbor technique and calculated the prediction accuracy for a group of databases from the UCI repository. Based on the experimental results of our study, we can state that, in general, it is not possible to know a priori a specific value of k to correctly classify an unseen example. Keywords: Nearest Neighbor, Local Adaptive Nearest Neighbor. 1 Introduction In Supervised Learning, systems based on examples (CBR, Case Based Rea- soning) have been object of study and improvement from their introduction at the end of the fifties. These algorithms extract knowledge through inductive processes from the partial descriptions given by the initial set of examples or instances. Machine learning process is usually accomplished in two functionally different phases. In the first phase of Training a model of the hyperspace is cre- ated by the labelled examples. In the second phase of Classification the new examples are labelled based on the constructed model. In the Nearest Neighbor algorithm (from here on NN) the training examples are the model itself. NN assigns to each new query the label of its nearest neighbor among those that are remembered from the phase of Training. In order to improve the accuracy with noise present in data, the k-NN algorithm introduces a parameter k so that for each new example q to  be classified the classes of the k nearest neighbors of q are considered: q will be labelled with the majority class or, in case of tie, it is randomly broken. Another alternative consists in assigning that class whose aver- age distance is the smallest one or introducing a heuristically obtained threshold kl < k so that the assigned class will be that with a number of associated exam- ples greater than this threshold [lo]. Extending the classification criterion, the k-NN,, algorithms (Nearest Neighbor Weighted Voted) assign weights to the prediction made by each example. These weights can be inversely proportional to the distance with respect to  the example to be classified [4,6]. Therefore, the P.M.A. Sloot et al. (Eds.): ICCS 2003, LNCS 2658, pp. 766−773, 2003. Springer-Verlag Berlin Heidelberg 2003Empirical Evaluation of the Difficulty of Finding aGoodValue of k for the Nearest NeighborFig. 1. Horse Colic database. If the new example to be classified is a central point, the k value slightly determines the assigned label. Fig. 2. Horse Colic database. If the new query is a border point, the k value can be decisive in the classi- ficat ion. k number of examples observed and the metric used t o  classify a test example are decisive parameters. Usually k is heuristically determined by the user or by means of cross-validation [9]. The usual metrics of these algorithms are the Eu- clidean distance for continuous attributes and the Overlap distance for nominal attributes (both metrics were used in our experiments). In the last years have appeared interesting approaches that test new metrics [13] or new data representations [2] t o  improve accuracy and computational com- plexity. Nevertheless, in spite of having a wide and diverse field of application, to  determine with certainty when k-NNobtains higher accuracy than NN[l ]  and viceversa [8] is still an open problem. In [5] it was proven that when the distance among examples with the same class is smaller than the distance among exam- ples of different class, the probability of error for NN and k-NN tends to  0 and i, respectively. But, not always this distribution for input data appears, reason why k-NN and k-NN,, can improve the results given by NN with noise present in the data. In [12] a study of the different situations in which k-NN improves the results of NNis exposed, and four classifiers are proposed (Locally Adaptive Nearest Neighbor, localKNNk,) where for each new example q to  be classified the parameter k takes a value k ,  which is near to  the values that classified the M (an extra parameter) nearest neighbors e, of q. In this work we intend t o  study the limits that the k-NN algorithm presents even when the value of k is not fixed but variable for each example. When an example as the Figure 1 illustrates is interior t o  a region of examples with its same label (it is a central point), the assigned label will depend little on the value of k. However, with an example near the decision boundaries (a  border point, see Figure 2) the choice such parameter can be decisive. In the following section we explain several results obtained after applying the standard and weighted k-Nearest Neighbor algorithm (from now k-NN and k-NN,,) with databases from the UCI repository [3]. In principle it seems logical t o  consider that the classification accuracy can improve when the k value is adjusted locally. In a previous work [7] we introduced a local nearest neighbor classifier which evaluates 767Empirical Evaluation of the Difficulty of Finding a Good Value of k Table 1. Percentage of examples that are correctly classified by kNN and kNNw, where k is an odd number belongs to the interval [1,11]. several k values to  decide the label for a new query. But the results obtained in the really interesting domains were very similar to the results given by k-NN, so the added computational complexity (the calculation of this local k value) can not be worth. In the next empirical analysis we show that local classifiers based on geometric proximity present a limit for classifying very near the Nearest Neighbor prediction ability, and we try to find somehow an empirical measure for that limit. DB An Aud Aut BS BC CHD CR GC G1 HS He HC 10 Ir PD P T  Son Soy Ve Vot Vow Wi WBC Zoo Av. 2 Empirical Evaluation k- NN k=l k=3 k=5 k=7 k=9 k=ll 92.53 88.97 87.63 83.96 85.30 85.52 74.33 66.81 63.71 59.73 58.40 60.61 75.60 66.82 60.97 57.07 57.07 57.56 79.03 79.84 80.32 86.4 88.96 88.80 70.27 69.58 72.02 74.12 74.12 74.82 74.58 81.84 81.51 82.17 81.51 81.18 81.88 85.94 86.52 86.52 86.81 86.37 72.60 73.00 73.30 72.89 72.89 73.20 70.09 68.22 64.01 61.21 58.87 57.47 75.55 79.25 80.00 81.11 80.37 81.48 80.64 82.58 83.87 83.87 84.51 84.51 68.47 69.29 69.02 70.38 69.83 69.02 86.89 86.03 85.47 84.04 84.33 84.04 95.33 95.33 95.33 96.66 95.33 94.66 70.57 74.08 74.08 75.26 73.82 73.43 34.21 29.20 33.03 35.69 34.21 35.39 87.50 83.65 82.21 80.28 75.96 72.59 91.80 91.80 91.06 90.48 90.19 89.31 69.85 68.43 67.73 68.91 67.49 66.90 91.03 91.49 92.64 93.56 93.10 93.56 99.39 97.97 94.24 89.89 83.53 42.72 95.50 96.62 96.06 96.06 96.06 95.50 95.27 96.56 96.99 96.85 96.85 96.70 96.03 92.07 93.06 91.08 89.10 89.10 80.37 79.81 79.36 79.09 78.27 76.43 In first place we obtained the error rates by leave-one-out validation increasing the value of k. The chosen limits for the maximum values of k were three: 11, 31 and 51. The results obtained for the odd numbers in the interval [1,11] applying k-NN and k-NN,, are showed in Table 1. Table 2 shows the average values for k in [1,11], [1,31] and k in [1,51]. Observing both Tables we can state that 768 F.J. Ferrer-Troyano, J.S. Aguilar-Ruiz, and J.C. RiquelmeTable 2. Average percentage of examples that are classified by k-NN and k-NN,, with an odd value of k in the intervals [1,11], [1,31] and [1,51]. DB Anneal- Audiology- Autos- Balance-Scale+ Breast-Cancer Cleveland-HD+ Credit-Rating+ German-Credit Glass- Heart-Statlog+ Hepatitis Horse-Colic- Ionosphere- Iris Pima-Diabetes+ Primary-Tumor+ Sonar - Soybean- Vehicle- Vote Vowel- Wine Wisconsin-BC zoo- Averages the performance of both algorithms is very similar, although there is a slight tendency in favor of k-NN,, with regard to  the obtained accuracy. From Tables 1 and 2 we can also observe that some databases have a high difficulty to  be classified, for instance, Primary- Tumor or Glass. Aiming for obtaining a priori the best value of k for each example, we wonder: "what would the gain be if it is possible to  find such a value for k?" , i.e., "how many examples are correctly classified for some value of k?". In other words: "how many examples are not correctly classified for any value of k by the Nearest Neighbor algorithm?". That is, there is not a value of k for which most of the k nearest neighbors of an example has the same label as such an example. This value provides an inter- esting rate because it gives an error bound for the Nearest Neighbor algorithm, and generally, for any classification met hod based on geometric proximity. To answer this question, we measured for each example all those values of k (among 1 and 51) that classified it correctly. If there was not value of k which clas- sified a certain example correctly, this example was indicated as non-classifiable. 769Empirical Evaluation of the Difficulty of Finding a Good Value of k Table 3. Percentage of examples that are not able to be correctly classified by k-NN and k-NN,, for any k in the intervals [1,11], [1,31] and [1,51]. DB Anneal Audiology Autos Balance- Scale Breast-Cancer Cleveland-HD Credit-Rating German-Credit Glass Heart-Statlog Hepatitis Horse-Colic Ionosphere Iris Pima-Diabetes Primary- Tumol Sonar Soybean Vehicle Vote Vowel Wine Wisconsin-BC zoo Averages Table 3 indicates the percentage of non-classifiable points for both techniques according to the fixed limits. Let's observe Horse-Colic. The 17.11% of examples does not correctly classify with any value of k in [1,11], the 14.94% of examples does not correctly classify with any k in [1,31] and the 14.14% is not correctly classified with k in [1,51]. Thus, we can state that there is not significant differ- ence among the limits 31 and 51, as well as between k-NN and k-NN,,. From Table 3 a maximum bound of the classification ability of k-NN can be obtained, still knowing a priori the value of k. That is, although k-NN could adapt locally so that for each example to be classified, according to  the values of its attributes, we calculated the best k, the error rates given in Table 3 can not be avoided. However, there are some databases in which the improvement in the accuracy can be worth the computational effort (the calculation of that local k). So, taking again Horse-Colic, we can observe in Tables 1 and 3 that we would have an error rate of 14.14% instead of 30.77%, i.e., an improvement of around 770 F.J. Ferrer-Troyano, J.S. Aguilar-Ruiz, and J.C. RiquelmeTable 4. Percentage of examples that have at least a number cvk of common values of k which classify it correctly and classify its nearest neighbor correctly by means of k-NN, when k E [I, 511 and cvk E {1,3,5,7,9,11,31,51). Anneal Audiology Autos Balance-scale Breast-cancer Cleveland-HD Credit-rating German-credit Glass Heart-statlog Hepatitis Horse-colic Ionosphere Iris Pima-diabetes Primary-tumor Sonar Soybean Vehicle Vote Vowel Wine Wisconsin-BC zoo Averages 50%. In general, logically, the highest increment is given for those databases that we point out previously as difficult to be classified by means of k-NN. Related to our initial objective that was to  find a relationship among the attributes values of any example and some correct value of k to classify it cor- rectly, we chose two databases with a significant gain, Glass and Horse-Colic. Next, for each domain we built a new database, where the label of each example was replaced by the minimum value of k for which such example was correctly classified. Then different approaches were attempted to  predict the parameter Ic: lineal and quadratic regression through traditional statistical methods, C4.5 (where the leaves in the decision tree obtained are possible values of k, and the Nearest Neighbor algorithm itself in a similar form to Locally Adaptive NN methods [ll]. None of these techniques was able to improve the average error rate obtained by the standard k-NN by applying ten-folds cross-validation. Note that it is not necessary applying again the Nearest Neighbor algorithm to vali- 771Empirical Evaluation of the Difficulty of Finding a Good Value of k date this last prediction approach because for each point we had calculated if a certain k value gave a correct classification. In a second approach we consider that maybe the problem could be in the choice of the minimum k as the label of the database, due to  the possible rela- tionship between the original space of attributes and the k value could be formed by a set of different k values. These new values might not necessarily coincide with the smallest k. In order to  solve this problem and t o  obtain more exact information, a second database was built. In this new database, the label of each example was replaced with a set of values that classified correctly such ex- ample. Given the special features of these data sets (multi-labelled), we carried out different approaches through regressions (lineal, quadratic, and quotient of polynomials). In this manner, the adjustment was correct if for each point the value obtained by means of regression was some of the k-labels associated the point. However, like the previous case, this type of regressions presented some results that did not improve the basic Nearest Neighbor algorithm. Finally, in order t o  measure what extent reaches the relationship between the k obtained for each example and the region of the space where this example is located, it was calculated the number of common k values that classified an example and its nearest neighbor. The results are shown in Table 4. In this Table we can observe that, again for Horse-Colic, only the 70.65% of examples have at least a k value shared with its nearest neighbor. In addition, this percentage decreases quickly when increasing the requirement that the number of shared k values must be higher. This means that if we tried to  predict the k that classifies a point correctly according to the k that classified its nearest neighbors correctly, we would have a minimum error rate of 29.35%, that is, the points for which its nearest neighbor have not any value that classifies it correctly. It is important to  notice that the values in Table 4 provide a superior bound of the probability to  guess the parameter k in function of the nearest neighbor of an example, but it does not mean that this probability will be reached easily. In fact, we can observe that for the databases that we have denominated difficult, with 3 or 4 common values, the percentage is so low that it seems complicated t o  determine the correct k by means of the Nearest Neighbor. 3 Conclusions and Future Directions A priori, we could consider that the value of k to  classify a new query through the Nearest Neighbor must depend on the space region in which the such example is located. Thus, when the point is central, the value of k can be low, and when it is near to  the decision boundaries, such value must be higher. Nevertheless, after our study, we can conclude that it is not possible to  determine with certainty the relationship between the attribute values for a particular point and the values of k that classifies it correctly through k-NN. To reach this observation, different tests have been carried out on databases from the UCI repository trying to  establish which are the accuracy that k - N N  gives as classification method. In this sense, we infer that t o  find a space distribution of the values of k in the 772 F.J. Ferrer-Troyano, J.S. Aguilar-Ruiz, and J.C. Riquelmeattributes space is not an easy task. As sample, it is enough with verifying that the percentage of common values of k between a point and its nearest neighbor falls quickly and, therefore, the disposition in concrete regions of the values of k for a later correct estimation of the same one does not seem feasible. At least, by applying the traditional tools as regression, 1-NN or C4.5. For future works we are trying t o  predict the correct value of k for each point from the original search space using genetic programming, which provides a capacity for obtaining regressions that are not bound by previous conditions. Another possible approach that we are studying is to  consider the value of k based on the enemy instead of the nearest neighbor, since this can provide us a measurement of the proximity from a point t o  the decision bound of the region in which it is. The research was supported by the Spanish Research Agency CICYT under grant TIC2001-1143-C03-02. References 1. D. W. Aha, D. Kibler, and M. K. Albert. Instance-based learning algorithms. Machine Learning, 6:37-66, 1991. 2. S. Arya, D. M. Mount, N. S. Netanyahu, R. Silverman, and A. Wu. An opti- mal algorithm for nearest neighbor searching. In Proceedings of 5th ACM SIAM Symposium on discrete Algorithms, pages 573-582, 1994. 3. C. Blake and E. K. Merz. Uci repository of machine learning databases, 1998. 4. S. Cost and S. Salzberg. A weighted nearest neighbor algorithm for learning with symbolic features. Machine Learning, 10:57-78, 1993. 5. T. M. Cover and P. E. Hart. Nearest neighbor pattern classification. IEEE Trans- actions on Information Theory, IT-13(1):21-27, 1967. 6. S.A. Dudani. The distance-weighted k-nearest-neighbor rule. IEEE Transactions on Systems, Man and Cybernetics, SMC-6, 4:325-327, 1975. 7. F. J .  Ferrer, J .  S. Aguilar, and J.  C. Riquelme. Nonparametric nearest neigh- bor with local adaptation. In Proceedings of the loth Portuguese Conference on Artificial Intelligence, Porto, Portugal, December 2001. 8. R. C. Holte. Very simple classification rules perform well on most commonly used datasets. Machine learning, 11:63-91, 1993. 9. M. Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society B, 36:lll-147, 1974. 10. I. Tomek. An experiment with the edited nearest-neighbor rule. IEEE Transactions on Systems, Man and Cybernetics, 6(6):448-452, June 1976. 11. D. Wettschereck and T. G. Dietterich. An experimental comparison of nearest neighbor and nearest hyperrect angle algorithms. Machine Learning, 19(1) :5-28, 1995. 12. D. Wettschereck and T.G. Dietterich. Locally adaptive nearest neighbor algo- rithms. Advances in Neural Information Processing Systems, (6) : 184-191, 1994. 13. D. R. Wilson and T. R. Martinez. Improved heterogeneous distance functions. Journal of Artificial1 Intelligence Research, 6(1):1-34, 1997. 773Empirical Evaluation of the Difficulty of Finding a Good Value of k Acknowledgment.

https://idus.us.es/xmlui/bitstream/handle/11441/39230/Empirical%20evaluation.pdf?sequence=1&isAllowed=y

Empirical Evaluation of the Difficulty of Finding a Good Value of k for the Nearest Neighbor

Abstract

Similar works

Full text

Available Versions

Crossref

idUS. Depósito de Investigación Universidad de Sevilla