Cheaper and Better: Selecting Good Workers for Crowdsourcing

Crowdsourcing provides a popular paradigm for data collection at scale. We study the problem of selecting subsets of workers from a given worker pool to maximize the accuracy under a budget constraint. One natural question is whether we should hire as many workers as the budget allows, or restrict on a small number of top-quality workers. By theoretically analyzing the error rate of a typical setting in crowdsourcing, we frame the worker selection problem into a combinatorial optimization problem and propose an algorithm to solve it efficiently. Empirical results on both simulated and real-world datasets show that our algorithm is able to select a small number of high-quality workers, and performs as good as, sometimes even better than, the much larger crowds as the budget allows

Learning to Predict the Wisdom of Crowds

The problem of "approximating the crowd" is that of estimating the crowd's majority opinion by querying only a subset of it. Algorithms that approximate the crowd can intelligently stretch a limited budget for a crowdsourcing task. We present an algorithm, "CrowdSense," that works in an online fashion to dynamically sample subsets of labelers based on an exploration/exploitation criterion. The algorithm produces a weighted combination of a subset of the labelers' votes that approximates the crowd's opinion.Comment: Presented at Collective Intelligence conference, 2012 (arXiv:1204.2991

Crowdsourcing for smart engagement apps in an urban context : an explorative study

This paper elaborates on the first results of an ongoing living lab project on ‘smart’ city engagement and offers a theoretical, methodological and empirical contribution to the field of user-driven innovation by describing a crowdsourcing experiment conducted in collaboration with the city of Ghent (Flanders). Our presented living lab approach has a double goal. First, it wants to empower citizens by systematically transforming the relationship(s) between citizens and between citizens (as service users) and local city-related governmental institutes (as service providers) by offering smart city applications. Second, it has the ambition to go beyond reactively studying information systems as change agents and wants to pro-actively improve engineering systems that can contribute to the desired changes in city engagement. Supporting citizens as self-actuating sensors to open up more innovative ways of collecting data is an important boundary of the research within a living lab context. We aim for user-driven innovation by involving citizens in the co-production of new electronic public services. Therefore we choose to go through a co-design process (Sanders & Stappers, 2008) with citizens defining the smart engagement applications that most probably will be developed and implemented in a living lab setting. Today, various innovation companies and organizations envision a central role for the user when looking for innovations. The attention for participation of the user is growing since the 80’s, although that the meaning of the concept ‘participation’ is not stable. Different people have used ‘participation’ in a wide variety of different situations and the widespread use of the term has tended to mean that ‘participation’ is used to refer to a wide variety of different situations by different people (Pateman, 1972). Therefore some point to participation as an empty signifier (Carpentier, 2007). The history and origin (and radicalism) of the concept as related to power issues is fading away under the diversity of its different meanings. Recently different participative methods were developed and are used to learn about users and their needs. Some known user-centered methods within industry are working with living labs (Niitamo, Kulkki, Eriksson, & Hribernik, 2006) and crowdsourcing (Hudson-Smith, Batty, Crooks, & Milton, 2009). Although participative methods were initially mainly focused on handing over the power to the user, currently much more attention is given to usability of applications and market forecasting when in the context of user involvement or co-creation. The analysis of power relations is fading slowly away. In our research the notion of participation is used in two ways: as a political phrase, referring to users who are gaining more power and impact on societal changes, and as a practical phrase referring to the forecasting of the success of urban smart engagement apps. This paper is structured in four parts. The first part of the paper introduces the concepts of engagement and ‘smartness’. The second part of the paper introduces crowdsourcing and also elaborates on the related concepts of ‘Web 2.0”, ‘collective intelligence’ and ‘wisdom of crowds’. The third part of the paper describes our methodology, introduces the online crowdsourcing enabler ‘mijndigitaalideevoorgent’, and presents the first, preliminary results of our crowdscourcing experiment. The fourth and last part of the paper formulates a conclusion and discussion of the results