Machine Learning competitions such as the Netflix Prize have proven
reasonably successful as a method of "crowdsourcing" prediction tasks. But
these competitions have a number of weaknesses, particularly in the incentive
structure they create for the participants. We propose a new approach, called a
Crowdsourced Learning Mechanism, in which participants collaboratively "learn"
a hypothesis for a given prediction task. The approach draws heavily from the
concept of a prediction market, where traders bet on the likelihood of a future
event. In our framework, the mechanism continues to publish the current
hypothesis, and participants can modify this hypothesis by wagering on an
update. The critical incentive property is that a participant will profit an
amount that scales according to how much her update improves performance on a
released test set.Comment: Full version of the extended abstract which appeared in NIPS 201
