The Power of Localization for Efficiently Learning Linear Separators with Noise

We introduce a new approach for designing computationally efficient learning algorithms that are tolerant to noise, and demonstrate its effectiveness by designing algorithms with improved noise tolerance guarantees for learning linear separators. We consider both the malicious noise model and the adversarial label noise model. For malicious noise, where the adversary can corrupt both the label and the features, we provide a polynomial-time algorithm for learning linear separators in $\Re^d$ under isotropic log-concave distributions that can tolerate a nearly information-theoretically optimal noise rate of $\eta = \Omega(\epsilon)$. For the adversarial label noise model, where the distribution over the feature vectors is unchanged, and the overall probability of a noisy label is constrained to be at most $\eta$, we also give a polynomial-time algorithm for learning linear separators in $\Re^d$ under isotropic log-concave distributions that can handle a noise rate of $\eta = \Omega\left(\epsilon\right)$. We show that, in the active learning model, our algorithms achieve a label complexity whose dependence on the error parameter $\epsilon$ is polylogarithmic. This provides the first polynomial-time active learning algorithm for learning linear separators in the presence of malicious noise or adversarial label noise.Comment: Contains improved label complexity analysis communicated to us by Steve Hannek

Learning with online constraints : shifting concepts and active learning

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2006.Includes bibliographical references (p. 99-102).Many practical problems such as forecasting, real-time decision making, streaming data applications, and resource-constrained learning, can be modeled as learning with online constraints. This thesis is concerned with analyzing and designing algorithms for learning under the following online constraints: i) The algorithm has only sequential, or one-at-time, access to data. ii) The time and space complexity of the algorithm must not scale with the number of observations. We analyze learning with online constraints in a variety of settings, including active learning. The active learning model is applicable to any domain in which unlabeled data is easy to come by and there exists a (potentially difficult or expensive) mechanism by which to attain labels. First, we analyze a supervised learning framework in which no statistical assumptions are made about the sequence of observations, and algorithms are evaluated based on their regret, i.e. their relative prediction loss with respect to the hindsight-optimal algorithm in a comparator class. We derive a, lower bound on regret for a class of online learning algorithms designed to track shifting concepts in this framework. We apply an algorithm we provided in previous work, that avoids this lower bound, to an energy-management problem in wireless networks, and demonstrate this application in a network simulation.(cont.) Second, we analyze a supervised learning framework in which the observations are assumed to be iid, and algorithms are compared by the number of prediction mistakes made in reaching a target generalization error. We provide a lower bound on mistakes for Perceptron, a standard online learning algorithm, for this framework. We introduce a modification to Perceptron and show that it avoids this lower bound, and in fact attains the optimal mistake-complexity for this setting. Third, we motivate and analyze an online active learning framework. The observations are assumed to be iid, and algorithms are judged by the number of label queries to reach a target generalization error. Our lower bound applies to the active learning setting as well, as a lower bound on labels for Perceptron paired with any active learning rule. We provide a new online active learning algorithm that avoids the lower bound, and we upper bound its label-complexity. The upper bound is optimal and also bounds the algorithm's total errors (labeled and unlabeled). We analyze the algorithm further, yielding a label-complexity bound under relaxed assumptions. Using optical character recognition data, we empirically compare the new algorithm to an online active learning algorithm with data-dependent performance guarantees, as well as to the combined variants of these two algorithms.by Claire E. Monteleoni.Ph.D

Learning with Online Constraints: Shifting Concepts and Active Learning

PhD thesisMany practical problems such as forecasting, real-time decisionmaking, streaming data applications, and resource-constrainedlearning, can be modeled as learning with online constraints. Thisthesis is concerned with analyzing and designing algorithms forlearning under the following online constraints: 1) The algorithm hasonly sequential, or one-at-time, access to data. 2) The time andspace complexity of the algorithm must not scale with the number ofobservations. We analyze learning with online constraints in avariety of settings, including active learning. The active learningmodel is applicable to any domain in which unlabeled data is easy tocome by and there exists a (potentially difficult or expensive)mechanism by which to attain labels.First, we analyze a supervised learning framework in which nostatistical assumptions are made about the sequence of observations,and algorithms are evaluated based on their regret, i.e. theirrelative prediction loss with respect to the hindsight-optimalalgorithm in a comparator class. We derive a lower bound on regretfor a class of online learning algorithms designed to track shiftingconcepts in this framework. We apply an algorithm we provided inprevious work, that avoids this lower bound, to an energy-managementproblem in wireless networks, and demonstrate this application in anetwork simulation. Second, we analyze a supervised learning frameworkin which the observations are assumed to be iid, and algorithms arecompared by the number of prediction mistakes made in reaching atarget generalization error. We provide a lower bound on mistakes forPerceptron, a standard online learning algorithm, for this framework.We introduce a modification to Perceptron and show that it avoids thislower bound, and in fact attains the optimal mistake-complexity forthis setting.Third, we motivate and analyze an online active learning framework.The observations are assumed to be iid, and algorithms are judged bythe number of label queries to reach a target generalizationerror. Our lower bound applies to the active learning setting as well,as a lower bound on labels for Perceptron paired with any activelearning rule. We provide a new online active learning algorithm thatavoids the lower bound, and we upper bound its label-complexity. Theupper bound is optimal and also bounds the algorithm's total errors(labeled and unlabeled). We analyze the algorithm further, yielding alabel-complexity bound under relaxed assumptions. Using opticalcharacter recognition data, we empirically compare the new algorithmto an online active learning algorithm with data-dependent performanceguarantees, as well as to the combined variants of these twoalgorithms

Algorithms for Query-Efficient Active Learning

Recent decades have witnessed great success of machine learning, especially for tasks where large annotated datasets are available for training models. However, in many applications, raw data, such as images, are abundant, but annotations, such as descriptions of images, are scarce. Annotating data requires human effort and can be expensive. Consequently, one of the central problems in machine learning is how to train an accurate model with as few human annotations as possible. Active learning addresses this problem by bringing the annotator to work together with the learner in the learning process. In active learning, a learner can sequentially select examples and ask the annotator for labels, so that it may require fewer annotations if the learning algorithm avoids querying less informative examples.This dissertation focuses on designing provable query-efficient active learning algorithms. The main contributions are as follows. First, we study noise-tolerant active learning in the standard stream-based setting. We propose a computationally efficient algorithm for actively learning homogeneous halfspaces under bounded noise, and prove it achieves nearly optimal label complexity. Second, we theoretically investigate a novel interactive model where the annotator can not only return noisy labels, but also abstain from labeling. We propose an algorithm which utilizes abstention responses, and analyze its statistical consistency and query complexity under different conditions of the noise and abstention rate. Finally, we study how to utilize auxiliary datasets in active learning. We consider a scenario where the learner has access to a logged observational dataset where labeled examples are observed conditioned on a selection policy. We propose algorithms that effectively take advantage of both auxiliary datasets and active learning. We prove that these algorithms are statistically consistent, and achieve a lower label requirement than alternative methods theoretically and empirically