22,387 research outputs found
Task Selection for Bandit-Based Task Assignment in Heterogeneous Crowdsourcing
Task selection (picking an appropriate labeling task) and worker selection
(assigning the labeling task to a suitable worker) are two major challenges in
task assignment for crowdsourcing. Recently, worker selection has been
successfully addressed by the bandit-based task assignment (BBTA) method, while
task selection has not been thoroughly investigated yet. In this paper, we
experimentally compare several task selection strategies borrowed from active
learning literature, and show that the least confidence strategy significantly
improves the performance of task assignment in crowdsourcing.Comment: arXiv admin note: substantial text overlap with arXiv:1507.0580
A Meta-Learning Approach to One-Step Active Learning
We consider the problem of learning when obtaining the training labels is
costly, which is usually tackled in the literature using active-learning
techniques. These approaches provide strategies to choose the examples to label
before or during training. These strategies are usually based on heuristics or
even theoretical measures, but are not learned as they are directly used during
training. We design a model which aims at \textit{learning active-learning
strategies} using a meta-learning setting. More specifically, we consider a
pool-based setting, where the system observes all the examples of the dataset
of a problem and has to choose the subset of examples to label in a single
shot. Experiments show encouraging results
Recommended from our members
Active learning of an action detector on untrimmed videos
textCollecting and annotating videos of realistic human actions is tedious, yet critical for training action recognition systems. We propose a method to actively request the most useful video annotations among a large set of unlabeled videos. Predicting the utility of annotating unlabeled video is not trivial, since any given clip may contain multiple actions of interest, and it need not be trimmed to temporal regions of interest. To deal with this problem, we propose a detection-based active learner to train action category models. We develop a voting-based framework to localize likely intervals of interest in an unlabeled clip, and use them to estimate the total reduction in uncertainty that annotating that clip would yield. On three datasets, we show our approach can learn accurate action detectors more efficiently than alternative active learning strategies that fail to accommodate the "untrimmed" nature of real video data.Computer Science
Learning to Sample: an Active Learning Framework
Meta-learning algorithms for active learning are emerging as a promising
paradigm for learning the ``best'' active learning strategy. However, current
learning-based active learning approaches still require sufficient training
data so as to generalize meta-learning models for active learning. This is
contrary to the nature of active learning which typically starts with a small
number of labeled samples. The unavailability of large amounts of labeled
samples for training meta-learning models would inevitably lead to poor
performance (e.g., instabilities and overfitting). In our paper, we tackle
these issues by proposing a novel learning-based active learning framework,
called Learning To Sample (LTS). This framework has two key components: a
sampling model and a boosting model, which can mutually learn from each other
in iterations to improve the performance of each other. Within this framework,
the sampling model incorporates uncertainty sampling and diversity sampling
into a unified process for optimization, enabling us to actively select the
most representative and informative samples based on an optimized integration
of uncertainty and diversity. To evaluate the effectiveness of the LTS
framework, we have conducted extensive experiments on three different
classification tasks: image classification, salary level prediction, and entity
resolution. The experimental results show that our LTS framework significantly
outperforms all the baselines when the label budget is limited, especially for
datasets with highly imbalanced classes. In addition to this, our LTS framework
can effectively tackle the cold start problem occurring in many existing active
learning approaches.Comment: Accepted by ICDM'1
An Entropy Search Portfolio for Bayesian Optimization
Bayesian optimization is a sample-efficient method for black-box global
optimization. How- ever, the performance of a Bayesian optimization method very
much depends on its exploration strategy, i.e. the choice of acquisition
function, and it is not clear a priori which choice will result in superior
performance. While portfolio methods provide an effective, principled way of
combining a collection of acquisition functions, they are often based on
measures of past performance which can be misleading. To address this issue, we
introduce the Entropy Search Portfolio (ESP): a novel approach to portfolio
construction which is motivated by information theoretic considerations. We
show that ESP outperforms existing portfolio methods on several real and
synthetic problems, including geostatistical datasets and simulated control
tasks. We not only show that ESP is able to offer performance as good as the
best, but unknown, acquisition function, but surprisingly it often gives better
performance. Finally, over a wide range of conditions we find that ESP is
robust to the inclusion of poor acquisition functions.Comment: 10 pages, 5 figure
- …