An Agricultural Harvest Knowledge Survey to Distinguish Types of Expertise

Gaining insight into the unique characteristics of participants during user research is a valuable tool for both recruitment and understanding differences within the target population. This work describes an agricultural harvest knowledge survey that was created for user research studies that observed experienced combine operators driving a combine simulator in virtual crop fields. Two variations of the survey were designed, utilized, and evaluated in two separate studies. Both studies found a difference between low and high knowledge operators\u27 performance on the knowledge survey in addition to performance differences. Based on the success of this survey as a population segmentation tool, the authors recommend three criteria for the design of future knowledge surveys in other domains: 1) use real world scenarios, 2) ensure question are neither too difficult nor too easy, and 3) ask the minimum number of questions to identify operator knowledge successfully. Future research aims to create a tool that can discern between system experts (with deep understanding of the system) and practice experts (who primarily have the wisdom of experience)

Fuzzy Logics for Multiple Choice Question Answering

We have recently witnessed how solutions based on neural-inspired architectures are the most popular in terms of Multiple-Choice Question Answering. However, solutions of this kind are difficult to interpret, require many resources for training, and present obstacles to transferring learning. In this work, we move away from this mainstream to explore new methods based on fuzzy logic that can cope with these problems. The results that can be obtained are in line with those of the neural cutting solutions, but with advantages such as their ease of interpretation, the low cost concerning the resources needed for training as well as the possibility of transferring the knowledge acquired in a much more straightforward and more intuitive way

Learning from Noisy Crowd Labels with Logics

This paper explores the integration of symbolic logic knowledge into deep neural networks for learning from noisy crowd labels. We introduce Logic-guided Learning from Noisy Crowd Labels (Logic-LNCL), an EM-alike iterative logic knowledge distillation framework that learns from both noisy labeled data and logic rules of interest. Unlike traditional EM methods, our framework contains a ``pseudo-E-step'' that distills from the logic rules a new type of learning target, which is then used in the ``pseudo-M-step'' for training the classifier. Extensive evaluations on two real-world datasets for text sentiment classification and named entity recognition demonstrate that the proposed framework improves the state-of-the-art and provides a new solution to learning from noisy crowd labels.Comment: 12 pages, 7 figures, accepted by ICDE-202

Learning from the Crowd with Pairwise Comparison

Efficient learning of halfspaces is arguably one of the most important problems in machine learning and statistics. With the unprecedented growth of large-scale data sets, it has become ubiquitous to appeal to crowd for data annotation, and the central problem that attracts a surge of recent interests is how one can provably learn the underlying halfspace from the highly noisy crowd feedback. On the other hand, a large body of recent works have been dedicated to the problem of learning with not only labels, but also pairwise comparisons, since in many cases it is easier to compare than to label. In this paper we study the problem of learning halfspaces from the crowd under the realizable PAC learning setting, and we assume that the crowd workers can provide (noisy) labels or pairwise comparison tags upon request. We show that with a powerful boosting framework, together with our novel design of a filtering process, the overhead (to be defined) of the crowd acts as a constant, whereas the natural extension of standard approaches to crowd setting leads to an overhead growing with the size of the data sets