Evidential uncertainties on rich labels for active learning

Recent research in active learning, and more precisely in uncertainty sampling, has focused on the decomposition of model uncertainty into reducible and irreducible uncertainties. In this paper, we propose to simplify the computational phase and remove the dependence on observations, but more importantly to take into account the uncertainty already present in the labels, \emph{i.e.} the uncertainty of the oracles. Two strategies are proposed, sampling by Klir uncertainty, which addresses the exploration-exploitation problem, and sampling by evidential epistemic uncertainty, which extends the reducible uncertainty to the evidential framework, both using the theory of belief functions

Real bird dataset with imprecise and uncertain values

The theory of belief functions allows the fusion of imperfect data from different sources. Unfortunately, few real, imprecise and uncertain datasets exist to test approaches using belief functions. We have built real birds datasets thanks to the collection of numerous human contributions that we make available to the scientific community. The interest of our datasets is that they are made of human contributions, thus the information is therefore naturally uncertain and imprecise. These imperfections are given directly by the persons. This article presents the data and their collection through crowdsourcing and how to obtain belief functions from the data

Real bird dataset with imprecise and uncertain values

International audienceThe theory of belief functions allows the fusion of imperfect data from different sources. Unfortunately, few real, imprecise and uncertain datasets exist to test approaches using belief functions. We have built real birds datasets thanks to the collection of numerous human contributions that we make available to the scientific community. The interest of our datasets is that they are made of human contributions, thus the information is therefore naturally uncertain and imprecise. These imperfections are given directly by the persons. This article presents the data and their collection through crowdsourcing and how to obtain belief functions from the data

Measuring the Expertise of Workers for Crowdsourcing Applications

International audienceCrowdsourcing platforms enable companies to propose tasks to a large crowd of users. The workers receive a compensation for their work according to the serious of the tasks they managed to accomplish. The evaluation of the quality of responses obtained from the crowd remains one of the most important problems in this context. Several methods have been proposed to estimate the expertise level of crowd workers. We propose an innovative measure of expertise assuming that we possess a dataset with an objective comparison of the items concerned. Our method is based on the definition of four factors with the theory of belief functions. We compare our method to the Fagin distance on a dataset from a real experiment, where users have to assess the quality of some audio recordings. Then, we propose to fuse both the Fagin distance and our expertise measure

Modélisation du profil des contributeurs dans les plateformes de crowdsourcing

International audienceThe crowdsourcing consists in the externalisation of tasks to a crowd of people remunerated to execute this ones. The crowd, usually diversified, can include users without qualification and/or motivation for the tasks. In this paper we will introduce a new method of user expertise modelization in the crowdsourcing platforms based on the theory of belief functions in order to identify serious and qualificated users.Le crowdsourcing consiste a l'externalisation de tâches à une foule de contributeurs rémunérés pour les effectuer. La foule, généralement très diversifiée, peut inclure des contributeurs non-qualifiés pour la tâche et/ou non-sérieux. Nous présentons ici une nouvelle méthode de modélisation de l'expertise du contributeur dans les plateformes de crowdsourcing se fondant sur la théorie des fonctions de croyance afin d'identifier les contributeurs sérieux et qualifiés

Imperfect Labels with Belief Functions for Active Learning

International audienceClassification is used to predict classes by extracting information from labeled data. But sometimes the collected data is imperfect, as in crowdsourcing where users have partial knowledge and may answer with uncertainty or imprecision. This paper offers a way to deal with uncertain and imprecise labeled data using Dempster-Shafer theory and active learning. An evidential version of K-NN that classifies a new example by observing its neighbors was earlier introduced. We propose to couple this approach with active learning, where the model uses only a fraction of the labeled data, and to compare it with non-evidential models. A new computable parameter for EK-NN is introduced, allowing the model to be both compatible with imperfectly labeled data and equivalent to its first version in the case of perfectly labeled data. This method increases the complexity but provides a way to work with imperfectly labeled data with efficient results and reduced labeling costs when coupled with active learning. We have conducted tests on real data imperfectly labeled during crowdsourcing campaigns

Estimation of the qualification and behavior of a contributor and aggregation of his answers in a crowdsourcing context

International audienceCrowdsourcing is the outsourcing of tasks to a crowd of contributors on a dedicated platform. The crowd on these platforms is very diversified and includes various profiles of contributors which generates data of uneven quality. However, majority voting, which is the aggregating method commonly used in platforms, gives equal weight to each contribution. To overcome this problem, we propose a method, MONITOR, which estimates the contributor's profile and aggregates the collected data by taking into account their possible imperfections thanks to the theory of belief functions. To do so, MONITOR starts by estimating the profile of the contributor through his qualification for the task and his behavior.Crowdsourcing campaigns have been carried out to collect the necessary data to test MONITOR on real data in order to compare it to existing approaches. The results of the experiments show that thanks to the use of the MONITOR method, we obtain a better rate of correct answer after aggregation of the contributions compared to the majority voting. Our contributions in this article are for the first time the proposal of a model that takes into account both the qualification of the contributor and his behavior in the estimation of his profile. For the second one, the weakening and the aggregation of the answers according to the estimated profiles

Validation of Smets' hypothesis in the crowdsourcing environment

International audienceIn the late 1990s, Philippe Smets hypothesizes that the more imprecise humans are, the more certain they are. The modeling of human responses by belief functions has been little discussed. In this context, it is essential to validate the hypothesis of Ph. Smets. This paper focuses on the experimental validation of this hypothesis in the context of crowdsourcing. Crowdsourcing is the outsourcing of tasks to users of dedicated platforms. Two crowdsourcing campaigns have been carried out. For the first one, the user could be imprecise in his answer, for the second one he had to be precise. For both experiments, the user had to indicate his certainty in his answer. The results show that by being imprecise, users are more certain of their answers.À la fin des années 1990, Philippe Smets émet l'hypothèse que plus les humains sont imprécis, plus ils sont certains. La modélisation des réponses humaines par des fonctions de croyance a été peu discutée. Dans ce contexte, il est essentiel de valider l'hypothèse de Ph. Smets. Cet article se concentre sur la validation expérimentale de cette hypothèse dans le contexte du crowdsourcing. Le crowdsourcing consiste à externaliser des tâches à des contributeurs sur des plateformes dédiées. Deux campagnes de crowdsourcing ont été réalisées. Pour la première, l'utilisateur pouvait être imprécis dans sa réponse, pour la seconde il devait être précis. Pour les deux expériences, l'utilisateur devait indiquer sa certitude dans sa réponse. Les résultats montrent qu'en étant imprécis, les utilisateurs sont plus certains de leurs réponses