7,966 research outputs found
An evaluation methodology for crowdsourced design
In recent years, the âpower of the crowdâ has been repeatedly demonstrated and various Internet platforms have been used to support applications of collaborative intelligence to tasks ranging from open innovation to image analysis. However, crowdsourcing applications in the fields of design research and creative innovation have been much slower to emerge. So, although there have been reports of systems and researchers using Internet crowdsourcing to carry out generative design, there are still many gaps in knowledge about the capability and limitations of the technology. Indeed the process models developed to support traditional commercial design (e.g. Pughâs Total Design, Agile, Double-Diamond etc.) have yet to be established for Crowdsourced Design. As a contribution to the development of such a general model this paper proposes the cDesign framework to support the creation of Crowdsourced Design activities. Within the cDesign framework the effective evaluation of design quality is identified as a key component that not only enables the leveraging of a large, virtual workforcesâ creative activities but is also fundamental to most iterative and optimisation processes. This paper reports an experimental investigation (developed using the cDesign framework) into two different Crowdsourced design evaluation approaches; free evaluation and âcrowdsourced Design Evaluation Criteriaâ (cDEC). The results are benchmarked against an evaluation carried out by a panel of experienced designers. The results suggest that the cDEC approach produces design rankings that correlate strongly with the judgements of an âexpert panelâ. The paper concludes that cDEC assessment methodology demonstrates how Crowdsourcing can be effectively used to evaluate, as well as generate, new design solutions
Social Turing Tests: Crowdsourcing Sybil Detection
As popular tools for spreading spam and malware, Sybils (or fake accounts)
pose a serious threat to online communities such as Online Social Networks
(OSNs). Today, sophisticated attackers are creating realistic Sybils that
effectively befriend legitimate users, rendering most automated Sybil detection
techniques ineffective. In this paper, we explore the feasibility of a
crowdsourced Sybil detection system for OSNs. We conduct a large user study on
the ability of humans to detect today's Sybil accounts, using a large corpus of
ground-truth Sybil accounts from the Facebook and Renren networks. We analyze
detection accuracy by both "experts" and "turkers" under a variety of
conditions, and find that while turkers vary significantly in their
effectiveness, experts consistently produce near-optimal results. We use these
results to drive the design of a multi-tier crowdsourcing Sybil detection
system. Using our user study data, we show that this system is scalable, and
can be highly effective either as a standalone system or as a complementary
technique to current tools
A goal model for crowdsourced software engineering
Crowdsourced Software Engineering (CSE) is the act of undertaking any external software engineering tasks by an undefined, potentially large group of online workers in an open call format. Using an open call, CSE recruits global online labor to work on various types of software engineering tasks, such as requirements extraction, design, coding and testing. The field is rising rapidly and touches various aspects of software engineering. CSE has grown significance in both academy and industry. Despite of the enormous usage and significance of CSE, there are many open challenges reported by various researchers. In order to
overcome the challenges and realizing the full potential of CSE, it is highly important to understand the concrete advantages and goals of CSE. In this paper, we present a goal model for CSE, to understand the real environment of CSE, and to explore the aspects that can somehow overcome the aforementioned challenges. The model is designed using RiSD, a method for building Strategic Dependency (SD) models in the i* notation, applied in this work using iStar2.0. This work can be considered useful for CSE stakeholders (Requesters, Workers, Platform owners and CSE organizations).Peer ReviewedPostprint (published version
Outsourcing labour to the cloud
Various forms of open sourcing to the online population are establishing themselves as cheap, effective methods of getting work done. These have revolutionised the traditional methods for innovation and have contributed to the enrichment of the concept of 'open innovation'. To date, the literature concerning this emerging topic has been spread across a diverse number of media, disciplines and academic journals. This paper attempts for the first time to survey the emerging phenomenon of open outsourcing of work to the internet using 'cloud computing'. The paper describes the volunteer origins and recent commercialisation of this business service. It then surveys the current platforms, applications and academic literature. Based on this, a generic classification for crowdsourcing tasks and a number of performance metrics are proposed. After discussing strengths and limitations, the paper concludes with an agenda for academic research in this new area
Learning Visual Importance for Graphic Designs and Data Visualizations
Knowing where people look and click on visual designs can provide clues about
how the designs are perceived, and where the most important or relevant content
lies. The most important content of a visual design can be used for effective
summarization or to facilitate retrieval from a database. We present automated
models that predict the relative importance of different elements in data
visualizations and graphic designs. Our models are neural networks trained on
human clicks and importance annotations on hundreds of designs. We collected a
new dataset of crowdsourced importance, and analyzed the predictions of our
models with respect to ground truth importance and human eye movements. We
demonstrate how such predictions of importance can be used for automatic design
retargeting and thumbnailing. User studies with hundreds of MTurk participants
validate that, with limited post-processing, our importance-driven applications
are on par with, or outperform, current state-of-the-art methods, including
natural image saliency. We also provide a demonstration of how our importance
predictions can be built into interactive design tools to offer immediate
feedback during the design process
Sustainability Standards and Stakeholder Engagement: Lessons From Carbon Markets
Stakeholders play an increasingly active role in private governance, including development of standards for measuring sustainability. Building on prior studies focused on standards and stakeholder engagement, we use an innovation management theoretical lens to compare stakeholder engagement and standards developed in two carbon markets: the Climate Action Reserve and the U.N.âs Clean Development Mechanism. We develop and test hypotheses regarding how different processes of stakeholder engagement in standard development affect the number, identity, and age of stakeholders involved, as well as the variation and quality of the resulting standards. In doing so, we contribute to the growing literature on stakeholder engagement in developing sustainability standards
- âŠ