29,828 research outputs found
Near-Optimally Teaching the Crowd to Classify
How should we present training examples to learners to teach them
classification rules? This is a natural problem when training workers for
crowdsourcing labeling tasks, and is also motivated by challenges in
data-driven online education. We propose a natural stochastic model of the
learners, modeling them as randomly switching among hypotheses based on
observed feedback. We then develop STRICT, an efficient algorithm for selecting
examples to teach to workers. Our solution greedily maximizes a submodular
surrogate objective function in order to select examples to show to the
learners. We prove that our strategy is competitive with the optimal teaching
policy. Moreover, for the special case of linear separators, we prove that an
exponential reduction in error probability can be achieved. Our experiments on
simulated workers as well as three real image annotation tasks on Amazon
Mechanical Turk show the effectiveness of our teaching algorithm
Learning Intelligent Dialogs for Bounding Box Annotation
We introduce Intelligent Annotation Dialogs for bounding box annotation. We
train an agent to automatically choose a sequence of actions for a human
annotator to produce a bounding box in a minimal amount of time. Specifically,
we consider two actions: box verification, where the annotator verifies a box
generated by an object detector, and manual box drawing. We explore two kinds
of agents, one based on predicting the probability that a box will be
positively verified, and the other based on reinforcement learning. We
demonstrate that (1) our agents are able to learn efficient annotation
strategies in several scenarios, automatically adapting to the image
difficulty, the desired quality of the boxes, and the detector strength; (2) in
all scenarios the resulting annotation dialogs speed up annotation compared to
manual box drawing alone and box verification alone, while also outperforming
any fixed combination of verification and drawing in most scenarios; (3) in a
realistic scenario where the detector is iteratively re-trained, our agents
evolve a series of strategies that reflect the shifting trade-off between
verification and drawing as the detector grows stronger.Comment: This paper appeared at CVPR 201
- …