26,483 research outputs found
Revisiting the Importance of Encoding Logic Rules in Sentiment Classification
We analyze the performance of different sentiment classification models on
syntactically complex inputs like A-but-B sentences. The first contribution of
this analysis addresses reproducible research: to meaningfully compare
different models, their accuracies must be averaged over far more random seeds
than what has traditionally been reported. With proper averaging in place, we
notice that the distillation model described in arXiv:1603.06318v4 [cs.LG],
which incorporates explicit logic rules for sentiment classification, is
ineffective. In contrast, using contextualized ELMo embeddings
(arXiv:1802.05365v2 [cs.CL]) instead of logic rules yields significantly better
performance. Additionally, we provide analysis and visualizations that
demonstrate ELMo's ability to implicitly learn logic rules. Finally, a
crowdsourced analysis reveals how ELMo outperforms baseline models even on
sentences with ambiguous sentiment labels.Comment: EMNLP 2018 Camera Read
Crowdsourcing Multiple Choice Science Questions
We present a novel method for obtaining high-quality, domain-targeted
multiple choice questions from crowd workers. Generating these questions can be
difficult without trading away originality, relevance or diversity in the
answer options. Our method addresses these problems by leveraging a large
corpus of domain-specific text and a small set of existing questions. It
produces model suggestions for document selection and answer distractor choice
which aid the human question generation process. With this method we have
assembled SciQ, a dataset of 13.7K multiple choice science exam questions
(Dataset available at http://allenai.org/data.html). We demonstrate that the
method produces in-domain questions by providing an analysis of this new
dataset and by showing that humans cannot distinguish the crowdsourced
questions from original questions. When using SciQ as additional training data
to existing questions, we observe accuracy improvements on real science exams.Comment: accepted for the Workshop on Noisy User-generated Text (W-NUT) 201
Reputation Agent: Prompting Fair Reviews in Gig Markets
Our study presents a new tool, Reputation Agent, to promote fairer reviews
from requesters (employers or customers) on gig markets. Unfair reviews,
created when requesters consider factors outside of a worker's control, are
known to plague gig workers and can result in lost job opportunities and even
termination from the marketplace. Our tool leverages machine learning to
implement an intelligent interface that: (1) uses deep learning to
automatically detect when an individual has included unfair factors into her
review (factors outside the worker's control per the policies of the market);
and (2) prompts the individual to reconsider her review if she has incorporated
unfair factors. To study the effectiveness of Reputation Agent, we conducted a
controlled experiment over different gig markets. Our experiment illustrates
that across markets, Reputation Agent, in contrast with traditional approaches,
motivates requesters to review gig workers' performance more fairly. We discuss
how tools that bring more transparency to employers about the policies of a gig
market can help build empathy thus resulting in reasoned discussions around
potential injustices towards workers generated by these interfaces. Our vision
is that with tools that promote truth and transparency we can bring fairer
treatment to gig workers.Comment: 12 pages, 5 figures, The Web Conference 2020, ACM WWW 202
- …