2,001 research outputs found
Analytic Methods for Optimizing Realtime Crowdsourcing
Realtime crowdsourcing research has demonstrated that it is possible to
recruit paid crowds within seconds by managing a small, fast-reacting worker
pool. Realtime crowds enable crowd-powered systems that respond at interactive
speeds: for example, cameras, robots and instant opinion polls. So far, these
techniques have mainly been proof-of-concept prototypes: research has not yet
attempted to understand how they might work at large scale or optimize their
cost/performance trade-offs. In this paper, we use queueing theory to analyze
the retainer model for realtime crowdsourcing, in particular its expected wait
time and cost to requesters. We provide an algorithm that allows requesters to
minimize their cost subject to performance requirements. We then propose and
analyze three techniques to improve performance: push notifications, shared
retainer pools, and precruitment, which involves recalling retainer workers
before a task actually arrives. An experimental validation finds that
precruited workers begin a task 500 milliseconds after it is posted, delivering
results below the one-second cognitive threshold for an end-user to stay in
flow.Comment: Presented at Collective Intelligence conference, 201
Information scraps: how and why information eludes our personal information management tools
In this paper we describe information scraps -- a class of personal information whose content is scribbled on Post-it notes, scrawled on corners of random sheets of paper, buried inside the bodies of e-mail messages sent to ourselves, or typed haphazardly into text files. Information scraps hold our great ideas, sketches, notes, reminders, driving directions, and even our poetry. We define information scraps to be the body of personal information that is held outside of its natural or We have much still to learn about these loose forms of information capture. Why are they so often held outside of our traditional PIM locations and instead on Post-its or in text files? Why must we sometimes go around our traditional PIM applications to hold on to our scraps, such as by e-mailing ourselves? What are information scraps' role in the larger space of personal information management, and what do they uniquely offer that we find so appealing? If these unorganized bits truly indicate the failure of our PIM tools, how might we begin to build better tools? We have pursued these questions by undertaking a study of 27 knowledge workers. In our findings we describe information scraps from several angles: their content, their location, and the factors that lead to their use, which we identify as ease of capture, flexibility of content and organization, and avilability at the time of need. We also consider the personal emotive responses around scrap management. We present a set of design considerations that we have derived from the analysis of our study results. We present our work on an application platform, jourknow, to test some of these design and usability findings
Do airstream mechanisms influence tongue movement paths?
Velar consonants often show an elliptical pattern of tongue movement in symmetrical vowel contexts, but the forces responsible for this remain unclear. We here consider the role of overpressure (increased intraoral air pressure) behind the constriction by examining how movement patterns are modified when speakers change from an egressive to ingressive airstream. Tongue movement and respiratory data were obtained from 3 speakers. The two airstream conditions were additionally combined with two levels of speech volume. The results showed consistent reductions in forward tongue movement during consonant closure in the ingressive conditions. Thus, overpressure behind the constriction may partly determine preferred movement patterns, but it cannot be the only influence since forward movement during closure is usually reduced but not eliminated in ingressive speech
Decremental All-Pairs ALL Shortest Paths and Betweenness Centrality
We consider the all pairs all shortest paths (APASP) problem, which maintains
the shortest path dag rooted at every vertex in a directed graph G=(V,E) with
positive edge weights. For this problem we present a decremental algorithm
(that supports the deletion of a vertex, or weight increases on edges incident
to a vertex). Our algorithm runs in amortized O(\vstar^2 \cdot \log n) time per
update, where n=|V|, and \vstar bounds the number of edges that lie on shortest
paths through any given vertex. Our APASP algorithm can be used for the
decremental computation of betweenness centrality (BC), a graph parameter that
is widely used in the analysis of large complex networks. No nontrivial
decremental algorithm for either problem was known prior to our work. Our
method is a generalization of the decremental algorithm of Demetrescu and
Italiano [DI04] for unique shortest paths, and for graphs with \vstar =O(n), we
match the bound in [DI04]. Thus for graphs with a constant number of shortest
paths between any pair of vertices, our algorithm maintains APASP and BC scores
in amortized time O(n^2 \log n) under decremental updates, regardless of the
number of edges in the graph.Comment: An extended abstract of this paper will appear in Proc. ISAAC 201
Globally Optimal Crowdsourcing Quality Management
We study crowdsourcing quality management, that is, given worker responses to
a set of tasks, our goal is to jointly estimate the true answers for the tasks,
as well as the quality of the workers. Prior work on this problem relies
primarily on applying Expectation-Maximization (EM) on the underlying maximum
likelihood problem to estimate true answers as well as worker quality.
Unfortunately, EM only provides a locally optimal solution rather than a
globally optimal one. Other solutions to the problem (that do not leverage EM)
fail to provide global optimality guarantees as well. In this paper, we focus
on filtering, where tasks require the evaluation of a yes/no predicate, and
rating, where tasks elicit integer scores from a finite domain. We design
algorithms for finding the global optimal estimates of correct task answers and
worker quality for the underlying maximum likelihood problem, and characterize
the complexity of these algorithms. Our algorithms conceptually consider all
mappings from tasks to true answers (typically a very large number), leveraging
two key ideas to reduce, by several orders of magnitude, the number of mappings
under consideration, while preserving optimality. We also demonstrate that
these algorithms often find more accurate estimates than EM-based algorithms.
This paper makes an important contribution towards understanding the inherent
complexity of globally optimal crowdsourcing quality management
Tie strength in question answer on social network sites
Asking friends, colleagues, or other trusted people to help answer a question or find information is a familiar and tried-and-true concept. Widespread use of online social networks has made social information seeking easier, and has provided researchers with opportunities to better observe this process. In this paper, we relate question answering to tie strength, a metric drawn from sociology describing how close a friendship is. We present a study evaluating the role of tie strength in question answers. We used previous research on tie strength in social media to generate tie strength information between participants and their answering friends, and asked them for feedback about the value of answers across several dimensions. While sociological studies have indicated that weak ties are able to provide better information, our findings are significant in that weak ties do not have this effect, and stronger ties (close friends) provide a subtle increase in information that contributes more to participants' overall knowledge, and is less likely to have been seen before
Speeding up shortest path algorithms
Given an arbitrary, non-negatively weighted, directed graph we
present an algorithm that computes all pairs shortest paths in time
, where is the number of
different edges contained in shortest paths and is a running
time of an algorithm to solve a single-source shortest path problem (SSSP).
This is a substantial improvement over a trivial times application of
that runs in . In our algorithm we use
as a black box and hence any improvement on results also in improvement
of our algorithm.
Furthermore, a combination of our method, Johnson's reweighting technique and
topological sorting results in an all-pairs
shortest path algorithm for arbitrarily-weighted directed acyclic graphs.
In addition, we also point out a connection between the complexity of a
certain sorting problem defined on shortest paths and SSSP.Comment: 10 page
Efficient crowdsourcing for multi-class labeling
Crowdsourcing systems like Amazon's Mechanical Turk have emerged as an effective large-scale human-powered platform for performing tasks in domains such as image classification, data entry, recommendation, and proofreading. Since workers are low-paid (a few cents per task) and tasks performed are monotonous, the answers obtained are noisy and hence unreliable. To obtain reliable estimates, it is essential to utilize appropriate inference algorithms (e.g. Majority voting) coupled with structured redundancy through task assignment. Our goal is to obtain the best possible trade-off between reliability and redundancy. In this paper, we consider a general probabilistic model for noisy observations for crowd-sourcing systems and pose the problem of minimizing the total price (i.e. redundancy) that must be paid to achieve a target overall reliability. Concretely, we show that it is possible to obtain an answer to each task correctly with probability 1-ε as long as the redundancy per task is O((K/q) log (K/ε)), where each task can have any of the distinct answers equally likely, q is the crowd-quality parameter that is defined through a probabilistic model. Further, effectively this is the best possible redundancy-accuracy trade-off any system design can achieve. Such a single-parameter crisp characterization of the (order-)optimal trade-off between redundancy and reliability has various useful operational consequences. Further, we analyze the robustness of our approach in the presence of adversarial workers and provide a bound on their influence on the redundancy-accuracy trade-off.
Unlike recent prior work [GKM11, KOS11, KOS11], our result applies to non-binary (i.e. K>2) tasks. In effect, we utilize algorithms for binary tasks (with inhomogeneous error model unlike that in [GKM11, KOS11, KOS11]) as key subroutine to obtain answers for K-ary tasks. Technically, the algorithm is based on low-rank approximation of weighted adjacency matrix for a random regular bipartite graph, weighted according to the answers provided by the workers.National Science Foundation (U.S.
Crowds in two seconds: Enabling realtime crowd-powered interfaces
Interactive systems must respond to user input within seconds. Therefore, to create realtime crowd-powered interfaces, we need to dramatically lower crowd latency. In this paper, we introduce the use of synchronous crowds for on-demand, realtime crowdsourcing. With synchronous crowds, systems can dynamically adapt tasks by leveraging the fact that workers are present at the same time. We develop techniques that recruit synchronous crowds in two seconds and use them to execute complex search tasks in ten seconds. The first technique, the retainer model, pays workers a small wage to wait and respond quickly when asked. We offer empirically derived guidelines for a retainer system that is low-cost and produces on-demand crowds in two seconds. Our second technique, rapid refinement, observes early signs of agreement in synchronous crowds and dynamically narrows the search space to focus on promising directions. This approach produces results that, on average, are of more reliable quality and arrive faster than the fastest crowd member working alone. To explore benefits and limitations of these techniques for interaction, we present three applications: Adrenaline, a crowd-powered camera where workers quickly filter a short video down to the best single moment for a photo; and Puppeteer and A|B, which examine creative generation tasks, communication with workers, and low-latency voting
Incentivizing High Quality Crowdwork
We study the causal effects of financial incentives on the quality of
crowdwork. We focus on performance-based payments (PBPs), bonus payments
awarded to workers for producing high quality work. We design and run
randomized behavioral experiments on the popular crowdsourcing platform Amazon
Mechanical Turk with the goal of understanding when, where, and why PBPs help,
identifying properties of the payment, payment structure, and the task itself
that make them most effective. We provide examples of tasks for which PBPs do
improve quality. For such tasks, the effectiveness of PBPs is not too sensitive
to the threshold for quality required to receive the bonus, while the magnitude
of the bonus must be large enough to make the reward salient. We also present
examples of tasks for which PBPs do not improve quality. Our results suggest
that for PBPs to improve quality, the task must be effort-responsive: the task
must allow workers to produce higher quality work by exerting more effort. We
also give a simple method to determine if a task is effort-responsive a priori.
Furthermore, our experiments suggest that all payments on Mechanical Turk are,
to some degree, implicitly performance-based in that workers believe their work
may be rejected if their performance is sufficiently poor. Finally, we propose
a new model of worker behavior that extends the standard principal-agent model
from economics to include a worker's subjective beliefs about his likelihood of
being paid, and show that the predictions of this model are in line with our
experimental findings. This model may be useful as a foundation for theoretical
studies of incentives in crowdsourcing markets.Comment: This is a preprint of an Article accepted for publication in WWW
\c{opyright} 2015 International World Wide Web Conference Committe
- …