2 research outputs found
Estimating Optimal Active Learning via Model Retraining Improvement
A central question for active learning (AL) is: "what is the optimal
selection?" Defining optimality by classifier loss produces a new
characterisation of optimal AL behaviour, by treating expected loss reduction
as a statistical target for estimation. This target forms the basis of model
retraining improvement (MRI), a novel approach providing a statistical
estimation framework for AL. This framework is constructed to address the
central question of AL optimality, and to motivate the design of estimation
algorithms. MRI allows the exploration of optimal AL behaviour, and the
examination of AL heuristics, showing precisely how they make sub-optimal
selections. The abstract formulation of MRI is used to provide a new guarantee
for AL, that an unbiased MRI estimator should outperform random selection. This
MRI framework reveals intricate estimation issues that in turn motivate the
construction of new statistical AL algorithms. One new algorithm in particular
performs strongly in a large-scale experimental study, compared to standard AL
methods. This competitive performance suggests that practical efforts to
minimise estimation bias may be important for AL applications.Comment: arXiv admin note: substantial text overlap with arXiv:1407.804
Understanding Goal-Oriented Active Learning via Influence Functions
Active learning (AL) concerns itself with learning a model from as few
labelled data as possible through actively and iteratively querying an oracle
with selected unlabelled samples. In this paper, we focus on analyzing a
popular type of AL in which the utility of a sample is measured by a specified
goal achieved by the retrained model after accounting for the sample's marginal
influence. Such AL strategies attract a lot of attention thanks to their
intuitive motivations, yet they also suffer from impractically high
computational costs due to their need for many iterations of model retraining.
With the help of influence functions, we present an effective approximation
that bypasses model retraining altogether, and propose a general efficient
implementation that makes such AL strategies applicable in practice, both in
the serial and the more challenging batch-mode setting. Additionally, we
present both theoretical and empirical findings which call into question a few
common practices and beliefs about such AL strategies.Comment: 14 pages, to be presented at the NeurIPS 2019 workshop on "ML with
Guarantees