102 research outputs found
Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection
Linguistically diverse datasets are critical for training and evaluating
robust machine learning systems, but data collection is a costly process that
often requires experts. Crowdsourcing the process of paraphrase generation is
an effective means of expanding natural language datasets, but there has been
limited analysis of the trade-offs that arise when designing tasks. In this
paper, we present the first systematic study of the key factors in
crowdsourcing paraphrase collection. We consider variations in instructions,
incentives, data domains, and workflows. We manually analyzed paraphrases for
correctness, grammaticality, and linguistic diversity. Our observations provide
new insight into the trade-offs between accuracy and diversity in crowd
responses that arise as a result of task design, providing guidance for future
paraphrase generation procedures.Comment: Published at ACL 201
Leaning left: a case study of media bias in the 2004 presidential election
No abstract available
Tuning the Diversity of Open-Ended Responses from the Crowd
Crowdsourcing can solve problems that current fully automated systems cannot.
Its effectiveness depends on the reliability, accuracy, and speed of the crowd
workers that drive it. These objectives are frequently at odds with one
another. For instance, how much time should workers be given to discover and
propose new solutions versus deliberate over those currently proposed? How do
we determine if discovering a new answer is appropriate at all? And how do we
manage workers who lack the expertise or attention needed to provide useful
input to a given task? We present a mechanism that uses distinct payoffs for
three possible worker actions---propose,vote, or abstain---to provide workers
with the necessary incentives to guarantee an effective (or even optimal)
balance between searching for new answers, assessing those currently available,
and, when they have insufficient expertise or insight for the task at hand,
abstaining. We provide a novel game theoretic analysis for this mechanism and
test it experimentally on an image---labeling problem and show that it allows a
system to reliably control the balance betweendiscovering new answers and
converging to existing ones
Noisy Embezzlement of Entanglement and Applications to Entanglement Dilution
In this thesis we present the concept of embezzlement of entanglement, its properties, efficiency, possible generalizations and propose the linear programming characterization of this phenomenon. Then, we focus on the noisy setting of embezzlement of entanglement. We provide the detailed proof of the quantum correlated sampling lemma which can be considered a protocol for noisy embezzlement of entanglement. Next, we propose a classical synchronization scheme for two spatially separated parties which do not communicate and use shared randomness to synchronize their descriptions of a quantum state. The result, together with the canonical embezzlement of entanglement, improves the quantum correlated sampling lemma for small quantum states in terms of the probability of success and distance between desired and final states. Then, we discuss the role of entanglement spread in dilution of entanglement. We propose an explicit protocol for the task of dilution of entanglement without communication. The protocol uses EPR pairs and an embezzling state of a relatively small size for the task of diluting entangled quantum states up to small infidelity. We modify the protocol to work in a noisy setting where the classical synchronization scheme finds its application
Reliable gains? Evidence for substantially underpowered designs in studies of working memory training transfer to fluid intelligence
In recent years, cognitive scientists and commercial interests (e.g., Fit Brains, Lumosity) have focused research attention and financial resources on cognitive tasks, especially working memory tasks, to explore and exploit possible transfer effects to general cognitive abilities, such as fluid intelligence. The increased research attention has produced mixed findings, as well as contention about the disposition of the evidence base. To address this contention, J. Au and colleagues (2014; doi:10.3758/s13423-014-0699-x) recently conducted a meta-analysis of extant controlled experimental studies of n-back task training transfer effects on measures of fluid intelligence in healthy adults; the results of which showed a small training transfer effect. Using several approaches, the current review evaluated and re-analyzed the meta-analytic data for the presence of two different forms of small-study effects: 1) publication bias in the presence of low power and; 2) low power in the absence of publication bias. The results of these approaches showed no evidence of selection bias in the working memory training literature, but did show evidence of small-study effects related to low power in the absence of publication bias. While the effect size estimate identified by Au and colleagues provided the most precise estimate to date, it should be interpreted in the context of a uniformly low-powered base of evidence. The present work concludes with a brief set of considerations for assessing the adequacy of a body of research findings for the application of meta-analytic techniques
The Effects of Sequence and Delay on Crowd Work
A common approach in crowdsourcing is to break large tasks into small microtasks so that they can be parallelized across many crowd workers and so that redundant work can be more easily compared for quality control. In practice, this can re-sult in the microtasks being presented out of their natural order and often introduces delays between individual micro-tasks. In this paper, we demonstrate in a study of 338 crowd workers that non-sequential microtasks and the introduction of delays significantly decreases worker performance. We show that interruptions where a large delay occurs between two related tasks can cause up to a 102 % slowdown in com-pletion time, and interruptions where workers are asked to perform different tasks in sequence can slow down comple-tion time by 57%. We conclude with a set of design guide-lines to improve both worker performance and realized pay, and instructions for implementing these changes in existing interfaces for crowd work. Author Keywords Crowdsourcing; human computation; workflows; continuity
- …