Skill-Aware Task Assignment in Crowdsourcing Applications

International audienceBesides simple human intelligence tasks such as image labeling, crowdsourcing platforms propose more and more tasks that require very specific skills. In such a setting we need to model skills that are required to execute a particular job. At the same time in order to match tasks to the crowd, we have to model the expertise of the participants. We present such a skill model that relies on a taxonomy. We also introduce task assignment algorithms to optimize the result quality. We illustrate the effectiveness of our algorithms and models through preliminary experiments with synthetic datasets

Engineering Crowdsourced Stream Processing Systems

A crowdsourced stream processing system (CSP) is a system that incorporates crowdsourced tasks in the processing of a data stream. This can be seen as enabling crowdsourcing work to be applied on a sample of large-scale data at high speed, or equivalently, enabling stream processing to employ human intelligence. It also leads to a substantial expansion of the capabilities of data processing systems. Engineering a CSP system requires the combination of human and machine computation elements. From a general systems theory perspective, this means taking into account inherited as well as emerging properties from both these elements. In this paper, we position CSP systems within a broader taxonomy, outline a series of design principles and evaluation metrics, present an extensible framework for their design, and describe several design patterns. We showcase the capabilities of CSP systems by performing a case study that applies our proposed framework to the design and analysis of a real system (AIDR) that classifies social media messages during time-critical crisis events. Results show that compared to a pure stream processing system, AIDR can achieve a higher data classification accuracy, while compared to a pure crowdsourcing solution, the system makes better use of human workers by requiring much less manual work effort

Hybrid human-AI driven open personalized education

Attaining those skills that match labor market demand is getting increasingly complicated as prerequisite knowledge, skills, and abilities are evolving dynamically through an uncontrollable and seemingly unpredictable process. Furthermore, people's interests in gaining knowledge pertaining to their personal life (e.g., hobbies and life-hacks) are also increasing dramatically in recent decades. In this situation, anticipating and addressing the learning needs are fundamental challenges to twenty-first century education. The need for such technologies has escalated due to the COVID-19 pandemic, where online education became a key player in all types of training programs. The burgeoning availability of data, not only on the demand side but also on the supply side (in the form of open/free educational resources) coupled with smart technologies, may provide a fertile ground for addressing this challenge. Therefore, this thesis aims to contribute to the literature about the utilization of (open and free-online) educational resources toward goal-driven personalized informal learning, by developing a novel Human-AI based system, called eDoer. In this thesis, we discuss all the new knowledge that was created in order to complete the system development, which includes 1) prototype development and qualitative user validation, 2) decomposing the preliminary requirements into meaningful components, 3) implementation and validation of each component, and 4) a final requirement analysis followed by combining the implemented components in order develop and validate the planned system (eDoer). All in all, our proposed system 1) derives the skill requirements for a wide range of occupations (as skills and jobs are typical goals in informal learning) through an analysis of online job vacancy announcements, 2) decomposes skills into learning topics, 3) collects a variety of open/free online educational resources that address those topics, 4) checks the quality of those resources and topic relevance using our developed intelligent prediction models, 5) helps learners to set their learning goals, 6) recommends personalized learning pathways and learning content based on individual learning goals, and 7) provides assessment services for learners to monitor their progress towards their desired learning objectives. Accordingly, we created a learning dashboard focusing on three Data Science related jobs and conducted an initial validation of eDoer through a randomized experiment. Controlling for the effects of prior knowledge as assessed by the pretest, the randomized experiment provided tentative support for the hypothesis that learners who engaged with personal eDoer recommendations attain higher scores on the posttest than those who did not. The hypothesis that learners who received personalized content in terms of format, length, level of detail, and content type, would achieve higher scores than those receiving non-personalized content was not supported as a statistically significant result

An ontology roadmap for crowdsourcing innovation intermediaries

Ontologies have proliferated in the last years, essentially justified by the need of achieving a consensus in the multiple representations of reality inside computers, and therefore the accomplishment of interoperability between machines and systems. Ontologies provide an explicit conceptualization that describes the semantics of the data. Crowdsourcing innovation intermediaries are organizations that mediate the communication and relationship between companies that aspire to solve some problem or to take advantage of any business opportunity with a crowd that is prone to give ideas based on their knowledge, experience and wisdom, taking advantage of web 2.0 tools. Various ontologies have emerged, but at the best of our knowledge, there isn’t any ontology that represents the entire process of intermediation of crowdsourcing innovation. In this paper we present an ontology roadmap for developing crowdsourcing innovation ontology of the intermediation process. Over the years, several authors have proposed some distinct methodologies, by different proposals of combining practices, activities, languages, according to the project they were involved in. We start making a literature review on ontology building, and analyse and compare ontologies that propose the development from scratch with the ones that propose reusing other ontologies. We also review enterprise and innovation ontologies known in literature. Finally, are presented the criteria for selecting the methodology and the roadmap for building crowdsourcing innovation intermediary ontology.(undefined

Ontology for Task and Quality Management in Crowdsourcing

This paper suggests an ontology for task and quality control mechanisms representation in crowdsourcing systems. The ontology is built to provide reasoning about tasks and quality control mechanisms to improve tasks and quality management in crowdsourcing. The ontology is formalized in OWL (Web Ontology Language) and implemented using Protégé. The developed ontology consists of 19 classes, 7 object properties, and 32 data properties. The development methodology of the ontology involves three phases including Specification (identifying scope, purpose and competency questions), Conceptualization (data dictionary, UML, and instance creation), and finally Implementation and Evaluation

Governing Actor Networks in an Emerging Crowdsourcing Ecosystem

Organisations harness the wisdom of community to solve problems or create new knowledge. Multiple organisations, diverse communities and multiple platforms are forming ecosystems to co-create value. We observe that Libraries, Archives, Galleries and Museums are forming collaborative crowdsourcing ecosystems to curate knowledge and create knowledge that ecosystem-wide stakeholders can use. However, despite the collaborative nature of crowdsourcing, various tensions arise among actors that hinder effective outcomes. Through a qualitative case study, we identify crowdsourcing actor networks and explore their tensions that hinder effective outcomes. We propose a strategic governance approach to foster crowdsourcing-based collaboration in a complex and dynamic ecosystem to create and capture value. This study presents a shift in the traditional schema of structured hierarchical governance of crowdsourcing projects to unstructured non-hierarchical governance of a multi-actor crowdsourcing ecosystem. The value propositions of crowdsourcing ecosystem actors networks are value co-creation, resource sharing, collective ownership, and mutual dependency

Feasibility investigation of crowdsourcing-based product design and development for manufacturing

In the era of Industry 4.0, to help manufacturers make quick response to rapidly changing market and customer needs, this research explores the feasibility of realizing benefits of crowdsourcing in product design and development from a lifecycle point of view through investigations on product design quality control and crowdsourcing technology theories, product design lifecycle information modelling, and simulation platform prototyping. It intends to help manufacturers create a product-service ecosystem to deliver values to all involved stakeholders of a PDD process. This study started with building up the theoretical foundation of product design quality control in crowdsourcing design environment. Then, key crowdsourcing technologies for realizing a lifecycle PDD process on a crowdsourcing platform while enabling the design quality were explored. Thirdly, a multi-layer product design lifecycle information model was developed to accommodate all design related information in a PDD process and the identified information at each design phase and the relationships and interactions among information entities were evaluated by case studies and ORM modelling method, respectively. Finally, two crowdsourcing platform prototypes based on the PDLIM were developed to test their effectiveness in communicating design information among stakeholders and delivering value to them. The proposed research made contributions to knowledge through the following improvements/advancements: (1) understanding of key factors affecting product design quality in crowdsourcing design environments, (2) a technical foundation of crowdsourcing technologies for PDD process, (3) a novel product design lifecycle information model accommodating design information in crowdsourcing environments, and (4) guidelines on developing intermediary and integrated crowdsourcing platforms for PDD