2,132 research outputs found
Much Ado About Time: Exhaustive Annotation of Temporal Data
Large-scale annotated datasets allow AI systems to learn from and build upon
the knowledge of the crowd. Many crowdsourcing techniques have been developed
for collecting image annotations. These techniques often implicitly rely on the
fact that a new input image takes a negligible amount of time to perceive. In
contrast, we investigate and determine the most cost-effective way of obtaining
high-quality multi-label annotations for temporal data such as videos. Watching
even a short 30-second video clip requires a significant time investment from a
crowd worker; thus, requesting multiple annotations following a single viewing
is an important cost-saving strategy. But how many questions should we ask per
video? We conclude that the optimal strategy is to ask as many questions as
possible in a HIT (up to 52 binary questions after watching a 30-second video
clip in our experiments). We demonstrate that while workers may not correctly
answer all questions, the cost-benefit analysis nevertheless favors consensus
from multiple such cheap-yet-imperfect iterations over more complex
alternatives. When compared with a one-question-per-video baseline, our method
is able to achieve a 10% improvement in recall 76.7% ours versus 66.7%
baseline) at comparable precision (83.8% ours versus 83.0% baseline) in about
half the annotation time (3.8 minutes ours compared to 7.1 minutes baseline).
We demonstrate the effectiveness of our method by collecting multi-label
annotations of 157 human activities on 1,815 videos.Comment: HCOMP 2016 Camera Read
A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality
Microtask crowdsourcing is increasingly critical to the creation of extremely
large datasets. As a result, crowd workers spend weeks or months repeating the
exact same tasks, making it necessary to understand their behavior over these
long periods of time. We utilize three large, longitudinal datasets of nine
million annotations collected from Amazon Mechanical Turk to examine claims
that workers fatigue or satisfice over these long periods, producing lower
quality work. We find that, contrary to these claims, workers are extremely
stable in their quality over the entire period. To understand whether workers
set their quality based on the task's requirements for acceptance, we then
perform an experiment where we vary the required quality for a large
crowdsourcing task. Workers did not adjust their quality based on the
acceptance threshold: workers who were above the threshold continued working at
their usual quality level, and workers below the threshold self-selected
themselves out of the task. Capitalizing on this consistency, we demonstrate
that it is possible to predict workers' long-term quality using just a glimpse
of their quality on the first five tasks.Comment: 10 pages, 11 figures, accepted CSCW 201
A Consumer-Centric Open Innovation Framework for Food and Packaging Manufacturing
This article has been archived following written permission from IGI Global.Closed innovation approaches have been employed for many years in the food industry. But, this sector recently perceives its end-user to be wary of radically new products and changes in consumption patterns. However, new product development involves not only the product itself but also the entire manufacturing and distribution network. In this paper, we present a new ICT based framework that embraces open innovation to place customers in the product development loop but at the same time assesses and eventually coordinates the entire manufacturing and supply chain. The aim is to design new food products that consumers will buy and at the same time ensure that these products will reach the consumer in time and at adequate quantity. On the product development side, our framework enables new food products that offer an integrated sensory experience of food and packaging, which encompass customization, healthy eating, and sustainability
Open collective innovation
Report published by Advanced Institute of Management ResearchThe innovation context is changing. The production of knowledge is accelerating.
Knowledge creation is now a globally distributed activity. Globalisation has massively
increased the range of markets and segments – putting pressure on innovation search
routines to cover much more territory. The proliferation of the internet and emergence
of large-scale social networks necessitates the development of new approaches
to innovation. The involvement of active users in innovation is accelerating.
As a result of the changing context in which innovation is taking place established
organisations need to review their approaches to innovation management.Economic and Social
Research Council (ESRC) and Engineering and Physical Sciences Research
Council (EPSRC
- …