719 research outputs found
Crowdsourcing in Computer Vision
Computer vision systems require large amounts of manually annotated data to
properly learn challenging visual concepts. Crowdsourcing platforms offer an
inexpensive method to capture human knowledge and understanding, for a vast
number of visual perception tasks. In this survey, we describe the types of
annotations computer vision researchers have collected using crowdsourcing, and
how they have ensured that this data is of high quality while annotation effort
is minimized. We begin by discussing data collection on both classic (e.g.,
object recognition) and recent (e.g., visual story-telling) vision tasks. We
then summarize key design decisions for creating effective data collection
interfaces and workflows, and present strategies for intelligently selecting
the most important data instances to annotate. Finally, we conclude with some
thoughts on the future of crowdsourcing in computer vision.Comment: A 69-page meta review of the field, Foundations and Trends in
Computer Graphics and Vision, 201
Human-in-the-Loop Learning From Crowdsourcing and Social Media
Computational social studies using public social media data have become more and more popular because of the large amount of user-generated data available. The richness of social media data, coupled with noise and subjectivity, raise significant challenges for computationally studying social issues in a feasible and scalable manner. Machine learning problems are, as a result, often subjective or ambiguous when humans are involved. That is, humans solving the same problems might come to legitimate but completely different conclusions, based on their personal experiences and beliefs. When building supervised learning models, particularly when using crowdsourced training data, multiple annotations per data item are usually reduced to a single label representing ground truth. This inevitably hides a rich source of diversity and subjectivity of opinions about the labels.
Label distribution learning associates for each data item a probability distribution over the labels for that item, thus it can preserve diversities of opinions, beliefs, etc. that conventional learning hides or ignores. We propose a humans-in-the-loop learning framework to model and study large volumes of unlabeled subjective social media data with less human effort. We study various annotation tasks given to crowdsourced annotators and methods for aggregating their contributions in a manner that preserves subjectivity and disagreement. We introduce a strategy for learning label distributions with only five-to-ten labels per item by aggregating human-annotated labels over multiple, semantically related data items. We conduct experiments using our learning framework on data related to two subjective social issues (work and employment, and suicide prevention) that touch many people worldwide. Our methods can be applied to a broad variety of problems, particularly social problems. Our experimental results suggest that specific label aggregation methods can help provide reliable representative semantics at the population level
On the evaluation and selection of classifier learning algorithms with crowdsourced data
In many current problems, the actual class of the instances, the ground truth, is unavail-
able. Instead, with the intention of learning a model, the labels can be crowdsourced by
harvesting them from different annotators. In this work, among those problems we fo-
cus on those that are binary classification problems. Specifically, our main objective is
to explore the evaluation and selection of models through the quantitative assessment of
the goodness of evaluation methods capable of dealing with this kind of context. That
is a key task for the selection of evaluation methods capable of performing a sensible
model selection. Regarding the evaluation and selection of models in such contexts,
we identify three general approaches, each one based on a different interpretation of
the nature of the underlying ground truth: deterministic, subjectivist or probabilistic.
For the analysis of these three approaches, we propose how to estimate the Area Under
the Curve (AUC) of the Receiver Operating Characteristic (ROC) curve within each
interpretation, thus deriving three evaluation methods. These methods are compared in
extensive experimentation whose empirical results show that the probabilistic method
generally overcomes the other two, as a result of which we conclude that it is advisable
to use that method when performing the evaluation in such contexts. In further studies,
it would be interesting to extend our research to multiclass classification problems
- …