Similarity Learning for Provably Accurate Sparse Linear Classification

In recent years, the crucial importance of metrics in machine learning algorithms has led to an increasing interest for optimizing distance and similarity functions. Most of the state of the art focus on learning Mahalanobis distances (requiring to fulfill a constraint of positive semi-definiteness) for use in a local k-NN algorithm. However, no theoretical link is established between the learned metrics and their performance in classification. In this paper, we make use of the formal framework of good similarities introduced by Balcan et al. to design an algorithm for learning a non PSD linear similarity optimized in a nonlinear feature space, which is then used to build a global linear classifier. We show that our approach has uniform stability and derive a generalization bound on the classification error. Experiments performed on various datasets confirm the effectiveness of our approach compared to state-of-the-art methods and provide evidence that (i) it is fast, (ii) robust to overfitting and (iii) produces very sparse classifiers.Comment: Appears in Proceedings of the 29th International Conference on Machine Learning (ICML 2012

Some results on PA-provably recursive functions

We provide some results which emerged from joint research carried out at CRM. The theorems are inspired by analogy with situations related to forcin

Phase Transitions for Gödel Incompleteness

Gödel's first incompleteness result from 1931 states that there are true assertions about the natural numbers which do not follow from the Peano axioms. Since 1931 many researchers have been looking for natural examples of such assertions and breakthroughs have been obtained in the seventies by Jeff Paris (in part jointly with Leo Harrington and Laurie Kirby) and Harvey Friedman who produced first mathematically interesting independence results in Ramsey theory (Paris) and well-order and well-quasi-order theory (Friedman). In this article we investigate Friedman style principles of combinatorial well-foundedness for the ordinals below epsilon_0. These principles state that there is a uniform bound on the length of decreasing sequences of ordinals which satisfy an elementary recursive growth rate condition with respect to their Gödel numbers. For these independence principles we classify (as a part of a general research program) their phase transitions, i.e. we classify exactly the bounding conditions which lead from provability to unprovability in the induced combinatorial well-foundedness principles. As Gödel numbering for ordinals we choose the one which is induced naturally from Gödel's coding of finite sequences from his classical 1931 paper on his incompleteness results. This choice makes the investigation highly non trivial but rewarding and we succeed in our objectives by using an intricate and surprising interplay between analytic combinatorics and the theory of descent recursive functions. For obtaining the required bounds on count functions for ordinals we use a classical 1961 Tauberian theorem by Parameswaran which apparently is far remote from Gödel's theorem

Recovering from Biased Data: Can Fairness Constraints Improve Accuracy?

Multiple fairness constraints have been proposed in the literature, motivated by a range of concerns about how demographic groups might be treated unfairly by machine learning classifiers. In this work we consider a different motivation; learning from biased training data. We posit several ways in which training data may be biased, including having a more noisy or negatively biased labeling process on members of a disadvantaged group, or a decreased prevalence of positive or negative examples from the disadvantaged group, or both. Given such biased training data, Empirical Risk Minimization (ERM) may produce a classifier that not only is biased but also has suboptimal accuracy on the true data distribution. We examine the ability of fairness-constrained ERM to correct this problem. In particular, we find that the Equal Opportunity fairness constraint [Hardt et al., 2016] combined with ERM will provably recover the Bayes optimal classifier under a range of bias models. We also consider other recovery methods including re-weighting the training data, Equalized Odds, and Demographic Parity, and Calibration. These theoretical results provide additional motivation for considering fairness interventions even if an actor cares primarily about accuracy

Unprovability and phase transitions in Ramsey theory

The first mathematically interesting, first-order arithmetical example of incompleteness was given in the late seventies and is know as the Paris-Harrington principle. It is a strengthened form of the finite Ramsey theorem which can not be proved, nor refuted in Peano Arithmetic. In this dissertation we investigate several other unprovable statements of Ramseyan nature and determine the threshold functions for the related phase transitions. Chapter 1 sketches out the historical development of unprovability and phase transitions, and offers a little information on Ramsey theory. In addition, it introduces the necessary mathematical background by giving definitions and some useful lemmas. Chapter 2 deals with the pigeonhole principle, presumably the most well-known, finite instance of the Ramsey theorem. Although straightforward in itself, the principle gives rise to unprovable statements. We investigate the related phase transitions and determine the threshold functions. Chapter 3 explores a phase transition related to the so-called infinite subsequence principle, which is another instance of Ramsey’s theorem. Chapter 4 considers the Ramsey theorem without restrictions on the dimensions and colours. First, generalisations of results on partitioning α-large sets are proved, as they are needed later. Second, we show that an iteration of a finite version of the Ramsey theorem leads to unprovability. Chapter 5 investigates the template “thin implies Ramsey”, of which one of the theorems of Nash-Williams is an example. After proving a more universal instance, we study the strength of the original Nash-Williams theorem. We conclude this chapter by presenting an unprovable statement related to Schreier families. Chapter 6 is intended as a vast introduction to the Atlas of prefixed polynomial equations. We begin with the necessary definitions, present some specific members of the Atlas, discuss several issues and give technical details