A general purpose conceptual model for crowdsourcing projects

Crowdsourcing is an approach that employs people to process input data to solve a computationally complex problem, such as generating a large dataset of annotated images, audio transcriptions or video scene descriptions. In this approach, people select tasks and produce individual results according to a list of steps that leads to an efficient solution. Then, every single result must be collected, interpreted, and integrated by a platform or system supporting the crowdsourcing process. This MSc dissertation starts with a state-of-the-art discussion, which provides an understanding of the main concepts and relationships reported in crowdsourcing projects found in the literature. By conducting a systematic review of crowdsourcing projects, we understand how these projects are designed and executed in the state-of-the-art, considering the following dimensions: Task execution, quality management, and platform usage. Our results summarized trends of the important aspects of a crowdsourcing project, such as crowd and task types, crowdsourcing platforms, and activities used to manage the quality; we also addressed functions and limitations in traditional crowdsourcing platforms, the definition of a crowdsourcing workflow, and the lack of standardization when designing a crowdsourcing project. In sequence, we developed a detailed conceptual model of crowdsourcing projects, specifying the essential entities and their relationships, based on the concepts leveraged in the accomplished systematic review. This work shows a class diagram that represents a general view of crowdsourcing projects, alongside with the concept of using an activity diagram to describe the execution of a specific project workflow, a neglected concept in previous works. To illustrate our contributions, the conceptual model is applied in some real crowdsourcing projects related to image annotation and segmentation data at scale, image QoE subjective assessments, software development, and cascading crowdsourcing to achieve complex video annotations

Novel Methods for Designing Tasks in Crowdsourcing

Crowdsourcing is becoming more popular as a means for scalable data processing that requires human intelligence. The involvement of groups of people to accomplish tasks could be an effective success factor for data-driven businesses. Unlike in other technical systems, the quality of the results depends on human factors and how well crowd workers understand the requirements of the task, to produce high-quality results. Looking at previous studies in this area, we found that one of the main factors that affect workers’ performance is the design of the crowdsourcing tasks. Previous studies of crowdsourcing task design covered a limited set of factors. The main contribution of this research is the focus on some of the less-studied technical factors, such as examining the effect of task ordering and class balance and measuring the consistency of the same task design over time and on different crowdsourcing platforms. Furthermore, this study ambitiously extends work towards understanding workers’ point of view in terms of the quality of the task and the payment aspect by performing a qualitative study with crowd workers and shedding light on some of the ethical issues around payments for crowdsourcing tasks. To achieve our goal, we performed several crowdsourcing experiments on specific platforms and measured the factors that influenced the quality of the overall result

Chatbots for Modelling, Modelling of Chatbots

Tesis Doctoral inédita leída en la Universidad Autónoma de Madrid, Escuela Politécnica Superior, Departamento de Ingeniería Informática. Fecha de Lectura: 28-03-202