Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions

Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar - all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.Comment: 40 pages main paper, 5 pages appendi

The Challenges of Knowledge Combination in ML-based Crowdsourcing – The ODF Killer Shrimp Challenge using ML and Kaggle

Organizations are increasingly using digital technologies, such as crowdsourcing platforms and machine learning, to tackle innovation challenges. These technologies often require the combination of heterogeneous technical and domain-specific knowledge from diverse actors to achieve the organization’s innovation goals. While research has focused on knowledge combination for relatively simple tasks on crowdsourcing platforms and within ML-based innovation, we know little about how knowledge is combined in emerging innovation approaches incorporating ML and crowdsourcing to solve domain-specific innovation challenges. Thus, this paper investigates the following: What are the challenges to knowledge combination in domain-specific ML-based crowdsourcing? We conducted a case study of an environmental challenge – how to use ML to predict the spread of a marine invasive species, led by the Swedish consortium, Ocean Data Factory Sweden using the crowdsourcing platform Kaggle. After discussing our results, we end the paper with recommendations on how to integrate crowdsourcing into domain-specific digital innovation processes

Cloud ConsultingCrowdsourcing-Based Framework for ERP Consulting

Information Technology (IT) altered the way of running businesses in all fields. The installation and integration of an Enterprise Resource Planning (ERP) system is a critical process. With the emergence of Web 2.0, many businesses were created or reengineered to gain an advantage from this technology. This offers an enormous network of users with different interests and skills, and support knowledge transfer. Our expectation is that ERP consulting business will be the next to be changed. In this work, a cloud consulting framework will be provided to enhance the services offered based on crowdsourcing. After examining literature on crowdsourcing and ERP consulting, the following research question was stated: “How can cloud consulting employ crowdsourcing to improve ERP consultancy?” Thus, the research objective is to develop a cloud consulting framework for ERP consulting based on crowdsourcing

Hybrid human-AI driven open personalized education

Attaining those skills that match labor market demand is getting increasingly complicated as prerequisite knowledge, skills, and abilities are evolving dynamically through an uncontrollable and seemingly unpredictable process. Furthermore, people's interests in gaining knowledge pertaining to their personal life (e.g., hobbies and life-hacks) are also increasing dramatically in recent decades. In this situation, anticipating and addressing the learning needs are fundamental challenges to twenty-first century education. The need for such technologies has escalated due to the COVID-19 pandemic, where online education became a key player in all types of training programs. The burgeoning availability of data, not only on the demand side but also on the supply side (in the form of open/free educational resources) coupled with smart technologies, may provide a fertile ground for addressing this challenge. Therefore, this thesis aims to contribute to the literature about the utilization of (open and free-online) educational resources toward goal-driven personalized informal learning, by developing a novel Human-AI based system, called eDoer. In this thesis, we discuss all the new knowledge that was created in order to complete the system development, which includes 1) prototype development and qualitative user validation, 2) decomposing the preliminary requirements into meaningful components, 3) implementation and validation of each component, and 4) a final requirement analysis followed by combining the implemented components in order develop and validate the planned system (eDoer). All in all, our proposed system 1) derives the skill requirements for a wide range of occupations (as skills and jobs are typical goals in informal learning) through an analysis of online job vacancy announcements, 2) decomposes skills into learning topics, 3) collects a variety of open/free online educational resources that address those topics, 4) checks the quality of those resources and topic relevance using our developed intelligent prediction models, 5) helps learners to set their learning goals, 6) recommends personalized learning pathways and learning content based on individual learning goals, and 7) provides assessment services for learners to monitor their progress towards their desired learning objectives. Accordingly, we created a learning dashboard focusing on three Data Science related jobs and conducted an initial validation of eDoer through a randomized experiment. Controlling for the effects of prior knowledge as assessed by the pretest, the randomized experiment provided tentative support for the hypothesis that learners who engaged with personal eDoer recommendations attain higher scores on the posttest than those who did not. The hypothesis that learners who received personalized content in terms of format, length, level of detail, and content type, would achieve higher scores than those receiving non-personalized content was not supported as a statistically significant result

Embracing open innovation to acquire external ideas and technologies and to transfer internal ideas and technologies outside

The objective of this dissertation is to increase understanding of how organizations can embrace open innovation in order to acquire external ideas and technologies from outside the organization, and to transfer internal ideas and technologies to outside the organization. The objective encompasses six sub-objectives, each addressed in one or more substudies. Altogether, the dissertation consists of nine substudies and a compendium summarizing the substudies. An extensive literature review was conducted on open innovation and crowdsourcing literature (substudies 1–4). In the subsequent empirical substudies, both qualitative research methods (substudies 5–7) and quantitative research methods (substudies 8–9) were applied. The four literature review substudies provided insights on the body of knowledge on open innovation and crowdsourcing. These substudies unveiled most of the influential articles, authors, and journals of open innovation and crowdsourcing disciplines. Moreover, they identified research gaps in the current literature. The empirical substudies offer several insightful findings. Substudy 5 shows how non-core ideas and technologies of a large firm can become valuable, especially for small firms. Intermediary platforms can find solutions to many pressing problems of large organizations by engaging renowned scientists from all over world (substudy 6). Intermediary platforms can also bring breakthrough innovations with novel mechanisms (substudy 7). Large firms are not only able to garner ideas by engaging their customers through crowdsourcing but they can also build long-lasting relations with their customers (substudies 8 and 9). Embracing open innovation brings challenges for firms too. Firms need to change their organizational structures in order to be able to fully benefit from open innovation. When crowdsourcing is successful, it produces a very large number of new ideas. This has the consequence that firms need to allocate a significant amount of resources in order to identify the most promising ideas. In an idea contest, customarily, only one or a few best ideas are rewarded (substudy 7). Sometimes, no reward is provided for the selected idea (substudies 8 and 9). Most of the ideas that are received are not implemented in practice