Clustering Without Knowing How To: Application and Evaluation

Crowdsourcing allows running simple human intelligence tasks on a large crowd of workers, enabling solving problems for which it is difficult to formulate an algorithm or train a machine learning model in reasonable time. One of such problems is data clustering by an under-specified criterion that is simple for humans, but difficult for machines. In this demonstration paper, we build a crowdsourced system for image clustering and release its code under a free license at https://github.com/Toloka/crowdclustering. Our experiments on two different image datasets, dresses from Zalando's FEIDEGGER and shoes from the Toloka Shoes Dataset, confirm that one can yield meaningful clusters with no machine learning algorithms purely with crowdsourcing.Comment: accepted at ECIR 2023 Demonstration Trac

Configuring Crowdsourcing for Requirements Elicitation

Crowdsourcing is an emerging paradigm which utilises the power of the crowd in contributing information and solving problems. Crowdsourcing can support requirements elicitation, especially for systems used by a wide range of users and working in a dynamic context where requirements evolve regularly. For such systems, traditional elicitation methods are typically costly and limited in catering for the high diversity, scale and volatility of requirements. In this paper, we advocate the use of crowdsourcing for requirements elicitation and investigate ways to configure crowdsourcing to improve the quality of elicited requirements. To confirm and enhance our argument, we follow an empirical approach starting with two focus groups involving 14 participants, users and developers, followed by an online expert survey involving 34 participants from the Requirements Engineering community. We discuss our findings and present a set of challenges of applying crowdsourcing to aid requirements engineering with a focus on the elicitation stage

Crowd-powered positive psychological interventions

Recent advances in crowdsourcing have led to new forms of assistive technologies, commonly referred to as crowd-powered devices. To best serve the user, these technologies crowdsource human intelligence as needed, when automated methods alone are insufficient. In this paper, we provide an overview of how these systems work and how they can be used to enhance technological interventions for positive psychology. As a specific example, we describe previous work that crowdsources positive reappraisals, providing users timely and personalized suggestions for ways to reconstrue stressful thoughts and situations. We then describe how this approach could be extended for use with other positive psychological interventions. Finally, we outline future directions for crowd-powered positive psychological interventions

The Value of Crowdsourcing for Complex Problems: Comparative Evidence from Software Developed By the Crowd And Professionals

Crowdsourcing is a problem solving model. In the context of complex problems, conventional theory suggests that solving complex problems is a province of professionals, that is, people with sufficient knowledge about the domain. Prior literature has indicated that the crowd, in addition to professionals, is also a great source for solving problems such as product innovation and idea generation. However, this assumption has yet to be tested. Adopting a quasi-experimental approach, this study uses a two-phase process to investigate this question. In the first phase we compare the development of a software by the crowd and professionals. In the second phase we evaluate the software developed by the crowdsourcing business model and professionals in terms of key perceived quality dimensions assessed by users of the systems. Quality is measured in terms of pragmatic quality, hedonic quality stimulation, and hedonic quality identification. Our study results suggest that there is a statistically significant difference between the software developed by a crowdsourcing business model and professionals in terms of hedonic quality stimulation and hedonic quality identification but there is no difference in terms of pragmatic quality. This research offers a first assessment of whether a crowdsourcing business model can be used to develop software with better user experience than professionallydeveloped software

A Gamification Framework for Sensor Data Analytics

The Internet of Things (IoT) enables connected objects to capture, communicate, and collect information over the network through a multitude of sensors, setting the foundation for applications such as smart grids, smart cars, and smart cities. In this context, large scale analytics is needed to extract knowledge and value from the data produced by these sensors. The ability to perform analytics on these data, however, is highly limited by the difficulties of collecting labels. Indeed, the machine learning techniques used to perform analytics rely upon data labels to learn and to validate results. Historically, crowdsourcing platforms have been used to gather labels, yet they cannot be directly used in the IoT because of poor human readability of sensor data. To overcome these limitations, this paper proposes a framework for sensor data analytics which leverages the power of crowdsourcing through gamification to acquire sensor data labels. The framework uses gamification as a socially engaging vehicle and as a way to motivate users to participate in various labelling tasks. To demonstrate the framework proposed, a case study is also presented. Evaluation results show the framework can successfully translate gamification events into sensor data labels