1,067 research outputs found
ConStance: Modeling Annotation Contexts to Improve Stance Classification
Manual annotations are a prerequisite for many applications of machine
learning. However, weaknesses in the annotation process itself are easy to
overlook. In particular, scholars often choose what information to give to
annotators without examining these decisions empirically. For subjective tasks
such as sentiment analysis, sarcasm, and stance detection, such choices can
impact results. Here, for the task of political stance detection on Twitter, we
show that providing too little context can result in noisy and uncertain
annotations, whereas providing too strong a context may cause it to outweigh
other signals. To characterize and reduce these biases, we develop ConStance, a
general model for reasoning about annotations across information conditions.
Given conflicting labels produced by multiple annotators seeing the same
instances with different contexts, ConStance simultaneously estimates gold
standard labels and also learns a classifier for new instances. We show that
the classifier learned by ConStance outperforms a variety of baselines at
predicting political stance, while the model's interpretable parameters shed
light on the effects of each context.Comment: To appear at EMNLP 201
A Margin-based MLE for Crowdsourced Partial Ranking
A preference order or ranking aggregated from pairwise comparison data is
commonly understood as a strict total order. However, in real-world scenarios,
some items are intrinsically ambiguous in comparisons, which may very well be
an inherent uncertainty of the data. In this case, the conventional total order
ranking can not capture such uncertainty with mere global ranking or utility
scores. In this paper, we are specifically interested in the recent surge in
crowdsourcing applications to predict partial but more accurate (i.e., making
less incorrect statements) orders rather than complete ones. To do so, we
propose a novel framework to learn some probabilistic models of partial orders
as a \emph{margin-based Maximum Likelihood Estimate} (MLE) method. We prove
that the induced MLE is a joint convex optimization problem with respect to all
the parameters, including the global ranking scores and margin parameter.
Moreover, three kinds of generalized linear models are studied, including the
basic uniform model, Bradley-Terry model, and Thurstone-Mosteller model,
equipped with some theoretical analysis on FDR and Power control for the
proposed methods. The validity of these models are supported by experiments
with both simulated and real-world datasets, which shows that the proposed
models exhibit improvements compared with traditional state-of-the-art
algorithms.Comment: 9 pages, Accepted by ACM Multimedia 2018 as a full pape
Crowdsourcing for Engineering Design: Objective Evaluations and Subjective Preferences
Crowdsourcing enables designers to reach out to large numbers of people who may not have been previously considered when designing a new product, listen to their input by aggregating their preferences and evaluations over potential designs, aiming to improve ``good'' and catch ``bad'' design decisions during the early-stage design process. This approach puts human designers--be they industrial designers, engineers, marketers, or executives--at the forefront, with computational crowdsourcing systems on the backend to aggregate subjective preferences (e.g., which next-generation Brand A design best competes stylistically with next-generation Brand B designs?) or objective evaluations (e.g., which military vehicle design has the best situational awareness?). These crowdsourcing aggregation systems are built using probabilistic approaches that account for the irrationality of human behavior (i.e., violations of reflexivity, symmetry, and transitivity), approximated by modern machine learning algorithms and optimization techniques as necessitated by the scale of data (millions of data points, hundreds of thousands of dimensions).
This dissertation presents research findings suggesting the unsuitability of current off-the-shelf crowdsourcing aggregation algorithms for real engineering design tasks due to the sparsity of expertise in the crowd, and methods that mitigate this limitation by incorporating appropriate information for expertise prediction. Next, we introduce and interpret a number of new probabilistic models for crowdsourced design to provide large-scale preference prediction and full design space generation, building on statistical and machine learning techniques such as sampling methods, variational inference, and deep representation learning. Finally, we show how these models and algorithms can advance crowdsourcing systems by abstracting away the underlying appropriate yet unwieldy mathematics, to easier-to-use visual interfaces practical for engineering design companies and governmental agencies engaged in complex engineering systems design.PhDDesign ScienceUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/133438/1/aburnap_1.pd
- …