Chen, Chen

Chernoff, Konstantin

Karemore, Gopal

Lauze, Francois Bernard

Lo, Pechin Chien Pau

Nielsen, Mads

English

Copenhagen University Research Information System

u n i ve r s i t y  o f  co pe n h ag e n  Københavns UniversitetNovel classification using adaptive metric k-NNChen, Chen; Chernoff, Konstantin; Karemore, Gopal; Lo, Pechin Chien Pau; Nielsen, Mads;Lauze, Francois BernardPublication date:2011Document versionPeer reviewed versionDocument license:UnspecifiedCitation for published version (APA):Chen, C., Chernoff, K., Karemore, G., Lo, P. C. P., Nielsen, M., & Lauze, F. B. (2011). Novel classification usingadaptive metric k-NN. Abstract from 1st IEEE Workshop on Kernels and Distances for Computer Vision inconjunction with ICCV 2011, Barcelona, Spain.Download date: 02. Feb. 2020Novel classification using adaptive metric k-NNC. Chen, K. Chernoff, G. Karemore, P. Lo, M. Nielsen, and F. LauzeDepartment of Computer Science, University of Copenhagen,Denmark1. INTRODUCTIONThe k-nearest-neighbors (k-NN) algorithm has been widely used as one of the simplest and model-free supervisedclassifier. It requires a distance function in feature space in order to identify the k nearest neighbors, and itsperformance is expected to be highly influenced by the specific choice of metric. The metric used is often chosenas the standard Euclidean metric, reflecting the absence of any specific prior on the data. However, especiallyfor high-dimensional data, this choice can contribute to the deterioration of the classifier, as it assumes thatthe data points are distributed in a hyperspherical manner in feature space and independent of the class labels,which might not always be optimal in terms of class-label discrimination.In recent years, it has been shown that using an appropriate metric can yield significant improvements whencompared to the Euclidean metric.1 In this work, we studied some special metrics, all from the simple class ofgeneralized Euclidean ones, defined by a symmetric (semi) positive definite (SPD or SSPD) matrix M on thefeature spacedM (x, y)2 = (x− y)tM(x− y). (1)The metrics under investigation are three Mahalanobis distance based metrics and a metric obtained throughsupervised learning. The first ones reflect typically some knowledge or a priori on the distribution of unlabeleddata, while the last one takes explicitly into account labeled data.2. ADAPTIVE METRIC k-NN2.1 Brownian Image ModelA well studied family of models for natural images are the so called Fractional Brownian Image Models introducedby Mandelbrot.2 According to Brownian Image Model theory, all the partial derivatives of a Brownian imageat any given location (x, y) are Gaussian distributed. Covariance of the outcome of the different filters can becomputed analytically:E[〈∂xm1yn1Gs1 ∗ I, ∂xm2yn2Gs2 ∗ I〉] = (−1)m+n2 +n2+m2σ202pin!m!2n+m(n+m)(n2 )!(m2 )!(2)whenever both m = m1 + m2 and n = n1 + n2 are even, otherwise the covariance is 0. The resulting metricdefined by Eq. (1) is with matrix MBIM = Σ−1BIM and we will refer to it as the “theoretical metric”.2.2 Normalized metricThe normalized metric situation is a very commonly used one, where one assumes that features are statisticallyindependent. The covariance matrix ΣN reduces to the diagonal matrix of variances of individual features. Thesevariances are computed empirically: (ΣN )ii = Var(xi), i = 1, . . . , N, where xi denotes the i-th component of theN -dimensional feature vector x. The resulting metric, as defined by Eq. (1), will be denoted as MN = Σ−1N . Wewill refer it as the “normalized metric” or “standardized metric”.2.3 Empirical metricWith Gaussian assumptions on data set, the covariance can be estimated via the classical empirical formula:ΣEMP =1UU∑i=u(xu − µ)(xu − µ)T , (3)where xu, u = 1, ..., U are sample points in the data set, U is the number of sample points, and µ is the empiricalmean of the data set. The resulting metric is referred as “empirical metric”, which reflects the structure of theunlabeled training data.0 0.2 0.4 0.6 0.8 100.10.20.30.40.50.60.70.80.91FPRTPRROC for CVD Data only using Gaussian derivative features   Euclidean metric:     0.8252Theoretical metric:   0.9135Empirical metric:     0.8984Normalized metric:  0.9220Optimized metric:    0.93130 0.2 0.4 0.6 0.8 100.10.20.30.40.50.60.70.80.91FPRTPRROC for CVD Data using all features  Euclidean metric:    0.7926Empirical metric:     0.8751Normalized metric:  0.9191Optimized metric:    0.9203Figure 1: The ROC curves and area under ROC curves of classification results using the standard Euclideanmetric and four adaptive metric (left: only using Gaussian derivative features; right: using all the features).2.4 Optimized metricAs opposed to the previous discussed cases, this section introduces a metric obtained through minimization of acost-functional corresponding to the weighted Leave-One-Group-Out (LOGO) error of k-NN.Given a sample set D, we assume it is naturally partitioned via “group” p, and let Dp denote the subset ofsamples that “belong to group” p and Dˆp its complement D\Dp. The u-th sample for group p is denoted x(u,p),and its label is denoted by l(u,p). The weighted k-NN LOGO error as:E(M ;D, k) =∑(u,p)k∑n=1wnδ(NNn(x(u,p), Dˆp), l(u,p)), (4)where NNn(x, S) is the label of n-th nearest neighbor of x, and wn is defined by∑kj=1 CjUξj(1 − ξ)U−j , whereξ = nU and U is the cardinality of the data set. The error is approximated by a differentiable smooth functionaland optimized using a stochastic version of the gradient descend algorithm.3. EXPERIMENT & RESULTWe applied the proposed methods on cardiovascular disease (CVD) data for detecting calcifications in the lumbarsection of the aorta in X-ray images. The data set consists of 14 images, and a total of 52 features at each of5000 points were extracted from each image: the image intensity, gradient magnitude, Hessian based features,and Gaussian derivatives at total order up to 3 at three different scales.Figure 1 shows the classification results of the Receiver Operating Characteristic (ROC) curves and the areaunder the ROC curves (numbers in the bottom boxes). As observed from the figure, all the proposed methodsperform better than the standard Euclidean metric. The optimized metric outperforms the rest. Among the restthree methods, which method performs the best depends on the training data.4. CONCLUSIONWe discussed adaptive metric k-NN for classification based on different prior knowledge: three Mahalanobisdistance based metrics and a optimized metric obtained via minimizing a k-NN specific classification error.Experiment on CVD data showed that k-NN classifier benefited greatly from the proposed metrics as comparedto the standard Euclidean one. The adaptive metric can be also applied to other distance based classifiers.REFERENCES1. E. P. Xing, A. Y. Ng, M. I. Jordan, and S. Russell, “Distance Metric Learning with Applications to Clusteringwith Side Information,” in Advances in Neural Information Processing Systems, 15, pp. 505–512, 2003.2. B. B. Mandelbrot, The Fractal Geometry of Nature, W. H. Freeman and Company, San Francisco, CA, 1982.

Novel classification using adaptive metric k-NN

https://curis.ku.dk/ws/files/46408790/chen_kdcv.pdf

Novel classification using adaptive metric k-NN

Abstract

Similar works

Full text

Available Versions

Copenhagen University Research Information System