A Framework for Exploring and Evaluating Mechanics in Human Computation Games

Human computation games (HCGs) are a crowdsourcing approach to solving computationally-intractable tasks using games. In this paper, we describe the need for generalizable HCG design knowledge that accommodates the needs of both players and tasks. We propose a formal representation of the mechanics in HCGs, providing a structural breakdown to visualize, compare, and explore the space of HCG mechanics. We present a methodology based on small-scale design experiments using fixed tasks while varying game elements to observe effects on both the player experience and the human computation task completion. Finally we discuss applications of our framework using comparisons of prior HCGs and recent design experiments. Ultimately, we wish to enable easier exploration and development of HCGs, helping these games provide meaningful player experiences while solving difficult problems.Comment: 11 pages, 5 figure

Crowdsourcing for Engineering Design: Objective Evaluations and Subjective Preferences

Crowdsourcing enables designers to reach out to large numbers of people who may not have been previously considered when designing a new product, listen to their input by aggregating their preferences and evaluations over potential designs, aiming to improve ``good'' and catch ``bad'' design decisions during the early-stage design process. This approach puts human designers--be they industrial designers, engineers, marketers, or executives--at the forefront, with computational crowdsourcing systems on the backend to aggregate subjective preferences (e.g., which next-generation Brand A design best competes stylistically with next-generation Brand B designs?) or objective evaluations (e.g., which military vehicle design has the best situational awareness?). These crowdsourcing aggregation systems are built using probabilistic approaches that account for the irrationality of human behavior (i.e., violations of reflexivity, symmetry, and transitivity), approximated by modern machine learning algorithms and optimization techniques as necessitated by the scale of data (millions of data points, hundreds of thousands of dimensions). This dissertation presents research findings suggesting the unsuitability of current off-the-shelf crowdsourcing aggregation algorithms for real engineering design tasks due to the sparsity of expertise in the crowd, and methods that mitigate this limitation by incorporating appropriate information for expertise prediction. Next, we introduce and interpret a number of new probabilistic models for crowdsourced design to provide large-scale preference prediction and full design space generation, building on statistical and machine learning techniques such as sampling methods, variational inference, and deep representation learning. Finally, we show how these models and algorithms can advance crowdsourcing systems by abstracting away the underlying appropriate yet unwieldy mathematics, to easier-to-use visual interfaces practical for engineering design companies and governmental agencies engaged in complex engineering systems design.PhDDesign ScienceUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/133438/1/aburnap_1.pd

Design Facets of Crowdsourcing

Crowdsourcing offers a way for information scientists to engage with the public and potentially collect valuable new data about documents. However, the space of crowdsourcing is very broad, with many design choices that differentiate existing projects significantly. This can make the space of crowdsourcing rather daunting. Building upon efforts from other fields to conceptualize, we develop a typology of crowdsourcing for information science. Through a number of dimensions within the scope of motivation, centrality, beneficiary, aggregation, type of work, and type of crowd, our typology provides a way to understand crowdsourcing.ye

A goal model for crowdsourced software engineering

Crowdsourced Software Engineering (CSE) is the act of undertaking any external software engineering tasks by an undefined, potentially large group of online workers in an open call format. Using an open call, CSE recruits global online labor to work on various types of software engineering tasks, such as requirements extraction, design, coding and testing. The field is rising rapidly and touches various aspects of software engineering. CSE has grown significance in both academy and industry. Despite of the enormous usage and significance of CSE, there are many open challenges reported by various researchers. In order to overcome the challenges and realizing the full potential of CSE, it is highly important to understand the concrete advantages and goals of CSE. In this paper, we present a goal model for CSE, to understand the real environment of CSE, and to explore the aspects that can somehow overcome the aforementioned challenges. The model is designed using RiSD, a method for building Strategic Dependency (SD) models in the i* notation, applied in this work using iStar2.0. This work can be considered useful for CSE stakeholders (Requesters, Workers, Platform owners and CSE organizations).Peer ReviewedPostprint (published version