3 research outputs found
Does Confidence Reporting from the Crowd Benefit Crowdsourcing Performance?
We explore the design of an effective crowdsourcing system for an -ary
classification task. Crowd workers complete simple binary microtasks whose
results are aggregated to give the final classification decision. We consider
the scenario where the workers have a reject option so that they are allowed to
skip microtasks when they are unable to or choose not to respond to binary
microtasks. Additionally, the workers report quantized confidence levels when
they are able to submit definitive answers. We present an aggregation approach
using a weighted majority voting rule, where each worker's response is assigned
an optimized weight to maximize crowd's classification performance. We obtain a
couterintuitive result that the classification performance does not benefit
from workers reporting quantized confidence. Therefore, the crowdsourcing
system designer should employ the reject option without requiring confidence
reporting.Comment: 6 pages, 4 figures, SocialSens 2017. arXiv admin note: text overlap
with arXiv:1602.0057
Mismatched Crowdsourcing based Language Perception for Under-resourced Languages
AbstractMismatched crowdsourcing is a technique for acquiring automatic speech recognizer training data in under-resourced languages by decoding the transcriptions of workers who don’t know the target language using a noisy-channel model of cross-language speech perception. All previous mismatched crowdsourcing studies have used English transcribers; this study is the first to recruit transcribers with a different native language, in this case, Mandarin Chinese. Using these data we are able to compute statistical models of cross-language perception of the tones and phonemes from transcribers based on phone distinctive features and tone features. By analyzing the phonetic and tonal variation mappings and coverages compared with the dictionary of the target language, we evaluate the different native languages’ effect on the transcribers’ performances