An Agricultural Harvest Knowledge Survey to Distinguish Types of Expertise

Gaining insight into the unique characteristics of participants during user research is a valuable tool for both recruitment and understanding differences within the target population. This work describes an agricultural harvest knowledge survey that was created for user research studies that observed experienced combine operators driving a combine simulator in virtual crop fields. Two variations of the survey were designed, utilized, and evaluated in two separate studies. Both studies found a difference between low and high knowledge operators\u27 performance on the knowledge survey in addition to performance differences. Based on the success of this survey as a population segmentation tool, the authors recommend three criteria for the design of future knowledge surveys in other domains: 1) use real world scenarios, 2) ensure question are neither too difficult nor too easy, and 3) ask the minimum number of questions to identify operator knowledge successfully. Future research aims to create a tool that can discern between system experts (with deep understanding of the system) and practice experts (who primarily have the wisdom of experience)

Cheaper and Better: Selecting Good Workers for Crowdsourcing

Crowdsourcing provides a popular paradigm for data collection at scale. We study the problem of selecting subsets of workers from a given worker pool to maximize the accuracy under a budget constraint. One natural question is whether we should hire as many workers as the budget allows, or restrict on a small number of top-quality workers. By theoretically analyzing the error rate of a typical setting in crowdsourcing, we frame the worker selection problem into a combinatorial optimization problem and propose an algorithm to solve it efficiently. Empirical results on both simulated and real-world datasets show that our algorithm is able to select a small number of high-quality workers, and performs as good as, sometimes even better than, the much larger crowds as the budget allows