Maximum Margin Multiclass Nearest Neighbors

We develop a general framework for margin-based multicategory classification in metric spaces. The basic work-horse is a margin-regularized version of the nearest-neighbor classifier. We prove generalization bounds that match the state of the art in sample size $n$ and significantly improve the dependence on the number of classes $k$. Our point of departure is a nearly Bayes-optimal finite-sample risk bound independent of $k$. Although $k$-free, this bound is unregularized and non-adaptive, which motivates our main result: Rademacher and scale-sensitive margin bounds with a logarithmic dependence on $k$. As the best previous risk estimates in this setting were of order $\sqrt k$, our bound is exponentially sharper. From the algorithmic standpoint, in doubling metric spaces our classifier may be trained on $n$ examples in $O(n^2\log n)$ time and evaluated on new points in $O(\log n)$ time

Laplacian Mixture Modeling for Network Analysis and Unsupervised Learning on Graphs

Laplacian mixture models identify overlapping regions of influence in unlabeled graph and network data in a scalable and computationally efficient way, yielding useful low-dimensional representations. By combining Laplacian eigenspace and finite mixture modeling methods, they provide probabilistic or fuzzy dimensionality reductions or domain decompositions for a variety of input data types, including mixture distributions, feature vectors, and graphs or networks. Provable optimal recovery using the algorithm is analytically shown for a nontrivial class of cluster graphs. Heuristic approximations for scalable high-performance implementations are described and empirically tested. Connections to PageRank and community detection in network analysis demonstrate the wide applicability of this approach. The origins of fuzzy spectral methods, beginning with generalized heat or diffusion equations in physics, are reviewed and summarized. Comparisons to other dimensionality reduction and clustering methods for challenging unsupervised machine learning problems are also discussed.Comment: 13 figures, 35 reference

Robustness Verification of k-Nearest Neighbor Classifiers by Abstract Interpretation

openAbstract interpretation is an established mathematical framework introduced by Cousot and Cousot in 1977 and ubiquitously used in static program analysis. In recent years, many noteworthy works have shown how abstract interpretation can be successfully applied to formally verify robustness properties of some major machine learning techniques like (deep) neural networks, decision trees and support vector machines. This research work aims to pursue this line of research by proposing a novel abstract interpretation-based framework for designing a sound abstract version of the k-Nearest Neighbors (kNN) algorithm, a well-known non-parametric supervised learning method widely used for classification and regression tasks, which is then instantiated to the standard interval domain approximating the range of numerical features, to verify its robustness and stability properties. This verification approach has been fully implemented and evaluated on several datasets, including standard benchmark datasets for individual fairness verification, and then compared with some related works finding adversarial examples on kNNs. The experimental results turned out to be very promising and showed high percentages of provable robustness and stability in most of the reference datasets, thus making a step forward in the current state-of-the-art of formal verification of machine learning models.Abstract interpretation is an established mathematical framework introduced by Cousot and Cousot in 1977 and ubiquitously used in static program analysis. In recent years, many noteworthy works have shown how abstract interpretation can be successfully applied to formally verify robustness properties of some major machine learning techniques like (deep) neural networks, decision trees and support vector machines. This research work aims to pursue this line of research by proposing a novel abstract interpretation-based framework for designing a sound abstract version of the k-Nearest Neighbors (kNN) algorithm, a well-known non-parametric supervised learning method widely used for classification and regression tasks, which is then instantiated to the standard interval domain approximating the range of numerical features, to verify its robustness and stability properties. This verification approach has been fully implemented and evaluated on several datasets, including standard benchmark datasets for individual fairness verification, and then compared with some related works finding adversarial examples on kNNs. The experimental results turned out to be very promising and showed high percentages of provable robustness and stability in most of the reference datasets, thus making a step forward in the current state-of-the-art of formal verification of machine learning models

Machine Learning

Machine Learning can be defined in various ways related to a scientific domain concerned with the design and development of theoretical and implementation tools that allow building systems with some Human Like intelligent behavior. Machine learning addresses more specifically the ability to improve automatically through experience

Distance Based Image Classification: A solution to generative classification's conundrum?

Most classifiers rely on discriminative boundaries that separate instances of each class from everything else. We argue that discriminative boundaries are counter-intuitive as they define semantics by what-they-are-not; and should be replaced by generative classifiers which define semantics by what-they-are. Unfortunately, generative classifiers are significantly less accurate. This may be caused by the tendency of generative models to focus on easy to model semantic generative factors and ignore non-semantic factors that are important but difficult to model. We propose a new generative model in which semantic factors are accommodated by shell theory's hierarchical generative process and non-semantic factors by an instance specific noise term. We use the model to develop a classification scheme which suppresses the impact of noise while preserving semantic cues. The result is a surprisingly accurate generative classifier, that takes the form of a modified nearest-neighbor algorithm; we term it distance classification. Unlike discriminative classifiers, a distance classifier: defines semantics by what-they-are; is amenable to incremental updates; and scales well with the number of classes.Comment: accepted by IJC

Efficient Classification for Metric Data

Recent advances in large-margin classification of data residing in general metric spaces (rather than Hilbert spaces) enable classification under various natural metrics, such as string edit and earthmover distance. A general framework developed for this purpose by von Luxburg and Bousquet [JMLR, 2004] left open the questions of computational efficiency and of providing direct bounds on generalization error. We design a new algorithm for classification in general metric spaces, whose runtime and accuracy depend on the doubling dimension of the data points, and can thus achieve superior classification performance in many common scenarios. The algorithmic core of our approach is an approximate (rather than exact) solution to the classical problems of Lipschitz extension and of Nearest Neighbor Search. The algorithm's generalization performance is guaranteed via the fat-shattering dimension of Lipschitz classifiers, and we present experimental evidence of its superiority to some common kernel methods. As a by-product, we offer a new perspective on the nearest neighbor classifier, which yields significantly sharper risk asymptotics than the classic analysis of Cover and Hart [IEEE Trans. Info. Theory, 1967].Comment: This is the full version of an extended abstract that appeared in Proceedings of the 23rd COLT, 201

Understanding deep learning

Deep neural networks have reached impressive performance in many tasks in computer vision and its applications. However, research into understanding deep neural networks is challenging due to the evaluation. Since it is unknown which features deep neural networks use, it is hard to empirically evaluate whether a result for which feature is used by a deep neural network is correct. The state- of-the-art for understanding which features a deep neural network uses to reach its prediction is sailiency maps. However, all methods built on sailiency maps share shortcomings that open a gap between the current state-of-the-art and the requirements for understanding deep neural networks. This work describes a method that does not suffer from these shortcomings. To this end, we employ the framework of causal modeling to determine whether a feature is used by the neural network. We present theoretical evidence that our method is able to correctly identify if a feature is used. Furthermore, we demonstrate two studies as empirical evidence. First, we show that our method can further the understanding of automatic skin lesion classifiers. There, we find that some of the features in the ABCD rule are used by the classifiers to identify melanoma but not to identify seborrheic keratosis. In contrast, all classifiers highly rely on the bias variables, particularly the age of the patient and the existence of colorful patches in the input image. Second we apply our method to adversarial debiasing. In adversarial debiasing, we want to stop a neural network from using a known bias variable. We demonstrate in a toy example and an example on real- world images that our approach outperforms the state-of-the-art in adversarial debiasing