Engineering Crowdsourced Stream Processing Systems

A crowdsourced stream processing system (CSP) is a system that incorporates crowdsourced tasks in the processing of a data stream. This can be seen as enabling crowdsourcing work to be applied on a sample of large-scale data at high speed, or equivalently, enabling stream processing to employ human intelligence. It also leads to a substantial expansion of the capabilities of data processing systems. Engineering a CSP system requires the combination of human and machine computation elements. From a general systems theory perspective, this means taking into account inherited as well as emerging properties from both these elements. In this paper, we position CSP systems within a broader taxonomy, outline a series of design principles and evaluation metrics, present an extensible framework for their design, and describe several design patterns. We showcase the capabilities of CSP systems by performing a case study that applies our proposed framework to the design and analysis of a real system (AIDR) that classifies social media messages during time-critical crisis events. Results show that compared to a pure stream processing system, AIDR can achieve a higher data classification accuracy, while compared to a pure crowdsourcing solution, the system makes better use of human workers by requiring much less manual work effort

A survey of the use of crowdsourcing in software engineering

The term 'crowdsourcing' was initially introduced in 2006 to describe an emerging distributed problem-solving model by online workers. Since then it has been widely studied and practiced to support software engineering. In this paper we provide a comprehensive survey of the use of crowdsourcing in software engineering, seeking to cover all literature on this topic. We first review the definitions of crowdsourcing and derive our definition of Crowdsourcing Software Engineering together with its taxonomy. Then we summarise industrial crowdsourcing practice in software engineering and corresponding case studies. We further analyse the software engineering domains, tasks and applications for crowdsourcing and the platforms and stakeholders involved in realising Crowdsourced Software Engineering solutions. We conclude by exposing trends, open issues and opportunities for future research on Crowdsourced Software Engineering

An ontology roadmap for crowdsourcing innovation intermediaries

Ontologies have proliferated in the last years, essentially justified by the need of achieving a consensus in the multiple representations of reality inside computers, and therefore the accomplishment of interoperability between machines and systems. Ontologies provide an explicit conceptualization that describes the semantics of the data. Crowdsourcing innovation intermediaries are organizations that mediate the communication and relationship between companies that aspire to solve some problem or to take advantage of any business opportunity with a crowd that is prone to give ideas based on their knowledge, experience and wisdom, taking advantage of web 2.0 tools. Various ontologies have emerged, but at the best of our knowledge, there isn’t any ontology that represents the entire process of intermediation of crowdsourcing innovation. In this paper we present an ontology roadmap for developing crowdsourcing innovation ontology of the intermediation process. Over the years, several authors have proposed some distinct methodologies, by different proposals of combining practices, activities, languages, according to the project they were involved in. We start making a literature review on ontology building, and analyse and compare ontologies that propose the development from scratch with the ones that propose reusing other ontologies. We also review enterprise and innovation ontologies known in literature. Finally, are presented the criteria for selecting the methodology and the roadmap for building crowdsourcing innovation intermediary ontology.(undefined

Waking Up a Sleeping Giant: Lessons from Two Extended Pilots to Transform Public Organizations by Internal Crowdsourcing

Digital transformation is a main driver for change, evolution, and disruption in organizations. As digital transformation is not solely determined by technological advancements, public environments necessitate changes in organizational practice and culture alike. A mechanism that seeks to realize employee engagement to adopt innovative modes of problem-solving is internal crowdsourcing, which flips the mode of operation from top-down to bottom-up. This concept is thus disrupting public organizations, as it heavily builds on IT-enabled engagement platforms that overcome the barriers of functional expertise and routine processes. Within this paper, we reflect on two design science projects that were piloted for six months within public organizations. We derive insights on the sociotechnical effects of internal crowdsourcing on organizational culture, social control, individual resources, motivation, and empowerment. Furthermore, using social cognitive theory, we propose design propositions for internal crowdsourcing, that guide future research and practice-oriented approaches to enable innovation in public organizations

A Framework for Quality Assurance in Crowdsourcing

The emergence of online paid micro-crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), allows on-demand and at scale distribution of tasks to human workers around the world. In such settings, online workers come and complete small tasks posted by a company, working for as long or as little as they wish. Such temporary employer-employee relationships give rise to adverse selection, moral hazard, and many other challenges. How can we ensure that the submitted work is accurate, especially when the verification cost is comparable to the cost of performing the task? How can we estimate the exhibited quality of the workers? What pricing strategies should be used to induce the effort of workers with varying ability levels? We develop a comprehensive framework for managing the quality in such micro crowdsourcing settings: First, we describe an algorithm for estimating the error rates of the participating workers, and show how to separate systematic worker biases from unrecoverable errors and generate an unbiased “worker quality” measurement. Next, we present a selective repeated-labeling algorithm that acquires labels in a way so that quality requirements can be met at minimum cost. Then, we propose a quality-adjusted pricing scheme that adjusts the payment level according to the contributed value by each worker. We test our compensation scheme in a principal-agent setting in which workers respond to incentives by varying their effort. Our simulation results demonstrate that the proposed pricing scheme is able to induce workers to exert higher levels of effort and yield larger profits for employers compared to the commonly adopted uniform pricing schemes. We also describe strategies that build on our quality control and pricing framework, to tackle crowdsourced tasks of increasingly higher complexity, while still maintaining a tight quality control of the process

Innovating Beyond the Fuzzy Front End: How to Use Reward-Based Crowdfunding to Co-create with Customers

Current research suggests that crowdfunding not only serves as an alternative source of capital but also as a flexible tool allowing start-ups to systematically integrate a crowd into their innovation processes. However, an adequate understanding of how start-ups can systematically leverage the co-creation potential of their early customers during crowdfunding is still nascent. Against this background, the aim of this research is to conceptualize and examine the concept of co-creation in the context of reward-based crowdfunding. In doing so, we distinguish it from other methods of user integration in the realm of open innovation and discuss how entrepreneurs can leverage reward-based crowdfunding to engage their customers in the development and deployment of their product and service offerings