Worker Skill Estimation from Crowdsourced Mutual Assessments

Current approaches for estimating skill levels of workforce either do not take into account the expertise of the recommender, or require intricate and expensive processes. In this paper, we propose a crowdsourcing algorithm for worker skill estimation based on mutual assessments. We propose a customized version of PageRank algorithm wherein we specifically considered the expertise of the person who made assessments. By implementing our algorithm on 15 real-world datasets from organizations and companies of varying sizes and domains and by using leave-one-out cross validation, we find that the results are highly correlated with the ground truth in datasets

T-Crowd: Effective Crowdsourcing for Tabular Data

Crowdsourcing employs human workers to solve computer-hard problems, such as data cleaning, entity resolution, and sentiment analysis. When crowdsourcing tabular data, e.g., the attribute values of an entity set, a worker's answers on the different attributes (e.g., the nationality and age of a celebrity star) are often treated independently. This assumption is not always true and can lead to suboptimal crowdsourcing performance. In this paper, we present the T-Crowd system, which takes into consideration the intricate relationships among tasks, in order to converge faster to their true values. Particularly, T-Crowd integrates each worker's answers on different attributes to effectively learn his/her trustworthiness and the true data values. The attribute relationship information is also used to guide task allocation to workers. Finally, T-Crowd seamlessly supports categorical and continuous attributes, which are the two main datatypes found in typical databases. Our extensive experiments on real and synthetic datasets show that T-Crowd outperforms state-of-the-art methods in terms of truth inference and reducing the cost of crowdsourcing

Knowledge‐Based Assignment Model for Allocation of Employees in Engineering‐to‐Order Production

In today’s rapidly changing business environment, it is necessary to react promptly in response to the product changes that happen constantly in an Engineering‐to‐Order production environment. Very often, there is not sufficient time to educate employees regarding new and necessary knowledge. If we insist on the standardization of a process execution, the process always requires appropriate knowledge from among available employees. In this chapter, an option for adjusting processes to available knowledge is studied. Following calculations, it was concluded that a partial corruption of a perfect process leads to a better knowledge alignment of employees. At first, with the corruption of a perfect process, its efficiency is decreased, but with better knowledge alignment, process efficiency is consequently increased to a level better than the original one. The optimization model presented in this chapter is based on a modified classic assignment problem and it includes a numerical example based on the data of ETO company. We proved our findings from the aspects of balance, employee capacity load and process efficiency

Efficiency, Sequenceability and Deal-Optimality in Fair Division of Indivisible Goods

National audienceIn fair division of indivisible goods, using sequences of sincere choices (or picking sequences) is a natural way to allocate the objects. The idea is as follows: at each stage, a designated agent picks one object among those that remain. Another intuitive way to obtain an allocation is to give objects to agents in the rst place, and to let agents exchange them as long as such "deals" are bene cial. This paper investigates these notions, when agents have additive preferences over objects, and unveils surprising connections between them, and with other e ciency and fairness notions. In particular, we show that an allocation is sequenceable if and only if it is optimal for a certain type of deals, namely cycle deals involving a single object. Furthermore, any Pareto-optimal allocation is sequenceable, but not the converse. Regarding fairness, we show that an allocation can be envy-free and non-sequenceable, but that every competitive equilibrium with equal incomes is sequenceable. To complete the picture, we show how some domain restrictions may a ect the relations between these notions. Finally, we experimentally explore the links between the scales of e ciency and fairness

HyTasker:Hybrid Task Allocation in Mobile Crowd Sensing

Task allocation is a major challenge in Mobile Crowd Sensing (MCS). While previous task allocation approaches follow either the opportunistic or participatory mode, this paper proposes to integrate these two complementary modes in a two-phased hybrid framework called HyTasker. In the offline phase, a group of workers (called opportunistic workers ) are selected, and they complete MCS tasks during their daily routines (i.e., opportunistic mode). In the online phase, we assign another set of workers (called participatory workers ) and require them to move specifically to perform tasks that are not completed by the opportunistic workers (i.e., participatory mode). Instead of considering these two phases separately, HyTasker jointly optimizes them with a total incentive budget constraint. In particular, when selecting opportunistic workers in the offline phase of HyTasker, we propose a novel algorithm that simultaneously considers the predicted task assignment for the participatory workers, in which the density and mobility of participatory workers are taken into account. Experiments on two real-world mobility datasets demonstrate that HyTasker outperforms other methods with more completed tasks under the same budget constraint

User Guidance for Efficient Fact Checking

The Web constitutes a valuable source of information. In recent years, it fostered the construction of large-scale knowledge bases, such as Freebase, YAGO, and DBpedia. The open nature of the Web, with content potentially being generated by everyone, however, leads to inaccuracies and misinformation. Construction and maintenance of a knowledge base thus has to rely on fact checking, an assessment of the credibility of facts. Due to an inherent lack of ground truth information, such fact checking cannot be done in a purely automated manner, but requires human involvement. In this paper, we propose a comprehensive framework to guide users in the validation of facts, striving for a minimisation of the invested effort. Our framework is grounded in a novel probabilistic model that combines user input with automated credibility inference. Based thereon, we show how to guide users in fact checking by identifying the facts for which validation is most beneficial. Moreover, our framework includes techniques to reduce the manual effort invested in fact checking by determining when to stop the validation and by supporting efficient batching strategies. We further show how to handle fact checking in a streaming setting. Our experiments with three real-world datasets demonstrate the efficiency and effectiveness of our framework: A knowledge base of high quality, with a precision of above 90\%, is constructed with only a half of the validation effort required by baseline techniques

Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions

Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar - all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.Comment: 40 pages main paper, 5 pages appendi