Modeling, enacting, and integrating custom crowdsourcing processes

Crowdsourcing (CS) is the outsourcing of a unit of work to a crowd of people via an open call for contributions. Thanks to the availability of online CS platforms, such as Amazon Mechanical Turk or CrowdFlower, the practice has experienced a tremendous growth over the past few years and demonstrated its viability in a variety of fields, such as data collection and analysis or human computation. Yet it is also increasingly struggling with the inherent limitations of these platforms: each platform has its own logic of how to crowdsource work (e.g., marketplace or contest), there is only very little support for structured work (work that requires the coordination of multiple tasks), and it is hard to integrate crowdsourced tasks into stateof-the-art business process management (BPM) or information systems. We attack these three shortcomings by (1) developing a flexible CS platform (we call it Crowd Computer, or CC) that allows one to program custom CS logics for individual and structured tasks, (2) devising a BPMN-based modeling language that allows one to program CC intuitively, (3) equipping the language with a dedicated visual editor, and (4) implementing CC on top of standard BPM technology that can easily be integrated into existing software and processes. We demonstrate the effectiveness of the approach with a case study on the crowd-based mining of mashup model patterns

A Collaboration Model for Community-Based Software Development with Social Machines

Crowdsourcing is generally used for tasks with minimal coordination, providing limited support for dynamic reconfiguration. Modern systems, exemplified by social ma chines, are subject to continual flux in both the client and development communities and their needs. To support crowdsourcing of open-ended development, systems must dynamically integrate human creativity with machine support. While workflows can be u sed to handle structured, predictable processes, they are less suitable for social machine development and its attendant uncertainty. We present models and techniques for coordination of human workers in crowdsourced software development environments. We combine the Social Compute Unit—a model of ad-hoc human worker teams—with versatile coordination protocols expressed in the Lightweight Social Calculus. This allows us to combine coordination and quality constraints with dynamic assessments of end-user desires, dynamically discovering and applying development protocols

Considering Human Aspects on Strategies for Designing and Managing Distributed Human Computation

A human computation system can be viewed as a distributed system in which the processors are humans, called workers. Such systems harness the cognitive power of a group of workers connected to the Internet to execute relatively simple tasks, whose solutions, once grouped, solve a problem that systems equipped with only machines could not solve satisfactorily. Examples of such systems are Amazon Mechanical Turk and the Zooniverse platform. A human computation application comprises a group of tasks, each of them can be performed by one worker. Tasks might have dependencies among each other. In this study, we propose a theoretical framework to analyze such type of application from a distributed systems point of view. Our framework is established on three dimensions that represent different perspectives in which human computation applications can be approached: quality-of-service requirements, design and management strategies, and human aspects. By using this framework, we review human computation in the perspective of programmers seeking to improve the design of human computation applications and managers seeking to increase the effectiveness of human computation infrastructures in running such applications. In doing so, besides integrating and organizing what has been done in this direction, we also put into perspective the fact that the human aspects of the workers in such systems introduce new challenges in terms of, for example, task assignment, dependency management, and fault prevention and tolerance. We discuss how they are related to distributed systems and other areas of knowledge.Comment: 3 figures, 1 tabl

Crowdy a framework for supporting socio-technical software : ecosystems with stream-based human computation

Ankara : The Department of Computer Engineering and the Graduate School of Engineering and Science of Bilkent University, 2014.Thesis (Master's) -- Bilkent University, 2014.Includes bibliographical references leaves 79-82.The scale of collaboration between people and computers has expanded leading to new era of computation called crowdsourcing. A variety of problems can be solved with this new approach by employing people to complete tasks that cannot be computerized. However, the existing approaches are focused on simplicity and independency of tasks that fall short to solve complex and sophisticated problems. We present Crowdy, a general-purpose and extensible crowdsourcing platform that lets users perform computations to solve complex problems using both computers and human workers. The platform is developed based on the stream-processing paradigm in which operators execute on the continuos stream of data elements. The proposed architecture provides a standard toolkit of operators for computation and configuration support to control and coordinate resources. There is no rigid structure or requirement that could limit the problem-set, which can be solved with the stream-based approach. The streambased human-computation approach is implemented and verified over different scenarios. Results show that sophisticated problems can be easily solved without significant amount of work for implementation. Also possible improvements are discussed and identified that is a promising future work for the existing work.Kalender, Mert EminM.S

Models and systems for managing sensor and crowd-oriented processes

Business process modeling refers to the design of business process models, using business processes languages, to orchestrate the work executed by employees, their interaction with external entities, and work items that are necessary to achieve a predefined goal. Model-driven development allows people, generally called modelers, to design also sophisticated application logic using high-level abstractions. Process modeling is typically connected with business, hence, existing process languages focus principally on the support and orchestration of activities executed by employees, or by external entities like web services. However, there is a wide range of other application logics that are process-driven and that can benefit from high-level abstractions to model low-level details. Our initial research focuses on distributed UIs, which are a distributed type of actors, and then particularly concentrated on Wireless Sensor Networks (WSNs) and crowdsourcing, which are distributed and also autonomous types of actors (they can execute a part of an application logic in an autonomous and isolated fashion). Developing applications in these areas requires a deep knowledge of the field and a non-trivial programming effort; domain experts have to code an orchestrate the logic executed by these actors. Since these applications are highly process-driven, domain experts could take advantage of high-level, process-oriented modeling conventions to design the internal logic of these kinds of applications. However, the intrinsic complexity of these domains and the current state of the art of modeling paradigms make the design and execution of processes for these new actors challenging. In this dissertation we analyze, design, and present modeling formalism and systems for managing processes in these contexts. We tackle the challenges of the three areas with an approach that analyzes and extends existing process modeling languages, to enable the design of the processes, and with an architecture, similar for the three focuses, to support the development and execution of processes. Starting from our initial work on the orchestration of distributed UIs, for which we present a modeling language with a set of modeling constructs specific for the UIs, we then present our contribution to WSNs and crowdsourcing domains, which are: a modeling convention for the development of WSN applications, with high-level modeling constructs that abstract the low-level details of the networks; and a modeling paradigm to design processes that are partially executed by a crowd of people. These languages are all equipped with prototypes that contain a modeling tool to design processes and a runtime environment to support the execution. The impact of this work is not only to the domains we focused on but also to the business process domain as we demonstrate how a process modeling is a flexible and suitable formalism to design processes with very diverging, domain-specific requirements

CrowdLang - First steps towards programmable human computers for general computation

Crowdsourcing markets such as Amazon’s Mechanical Turk provide an enormous potential for accomplishing work by combining human and machine computation. Today crowdsourcing is mostly used for massive parallel information processing for a variety of tasks such as image labeling. However, as we move to more sophisticated problem-solving there is little knowledge about managing dependencies between steps and a lack of tools for doing so. As the contribution of this paper, we present a concept of an executable, model-based programming language and a general purpose framework for accomplishing more sophisticated problems. Our approach is inspired by coordination theory and an analysis of emergent collective intelligence. We illustrate the applicability of our proposed language by combining machine and human computation based on existing interaction patterns for several general computation problems

Multi-objective Search-based Mobile Testing

Despite the tremendous popularity of mobile applications, mobile testing still relies heavily on manual testing. This thesis presents mobile test automation approaches based on multi-objective search. We introduce three approaches: Sapienz (for native Android app testing), Octopuz (for hybrid/web JavaScript app testing) and Polariz (for using crowdsourcing to support search-based mobile testing). These three approaches represent the primary scientific and technical contributions of the thesis. Since crowdsourcing is, itself, an emerging research area, and less well understood than search-based software engineering, the thesis also provides the first comprehensive survey on the use of crowdsourcing in software testing (in particular) and in software engineering (more generally). This survey represents a secondary contribution. Sapienz is an approach to Android testing that uses multi-objective search-based testing to automatically explore and optimise test sequences, minimising their length, while simultaneously maximising their coverage and fault revelation. The results of empirical studies demonstrate that Sapienz significantly outperforms both the state-of-the-art technique Dynodroid and the widely-used tool, Android Monkey, on all three objectives. When applied to the top 1,000 Google Play apps, Sapienz found 558 unique, previously unknown crashes. Octopuz reuses the Sapienz multi-objective search approach for automated JavaScript testing, aiming to investigate whether it replicates the Sapienz’ success on JavaScript testing. Experimental results on 10 real-world JavaScript apps provide evidence that Octopuz significantly outperforms the state of the art (and current state of practice) in automated JavaScript testing. Polariz is an approach that combines human (crowd) intelligence with machine (computational search) intelligence for mobile testing. It uses a platform that enables crowdsourced mobile testing from any source of app, via any terminal client, and by any crowd of workers. It generates replicable test scripts based on manual test traces produced by the crowd workforce, and automatically extracts from these test traces, motif events that can be used to improve search-based mobile testing approaches such as Sapienz