2,308 research outputs found
Partial Truthfulness in Minimal Peer Prediction Mechanisms with Limited Knowledge
We study minimal single-task peer prediction mechanisms that have limited
knowledge about agents' beliefs. Without knowing what agents' beliefs are or
eliciting additional information, it is not possible to design a truthful
mechanism in a Bayesian-Nash sense. We go beyond truthfulness and explore
equilibrium strategy profiles that are only partially truthful. Using the
results from the multi-armed bandit literature, we give a characterization of
how inefficient these equilibria are comparing to truthful reporting. We
measure the inefficiency of such strategies by counting the number of dishonest
reports that any minimal knowledge-bounded mechanism must have. We show that
the order of this number is , where is the number of
agents, and we provide a peer prediction mechanism that achieves this bound in
expectation
Measurement Integrity in Peer Prediction: A Peer Assessment Case Study
We propose measurement integrity, a property related to ex post reward
fairness, as a novel desideratum for peer prediction mechanisms in many natural
applications. Like robustness against strategic reporting, the property that
has been the primary focus of the peer prediction literature, measurement
integrity is an important consideration for understanding the practical
performance of peer prediction mechanisms. We perform computational
experiments, both with an agent-based model and with real data, to empirically
evaluate peer prediction mechanisms according to both of these important
properties. Our evaluations simulate the application of peer prediction
mechanisms to peer assessment -- a setting in which ex post fairness concerns
are particularly salient. We find that peer prediction mechanisms, as proposed
in the literature, largely fail to demonstrate significant measurement
integrity in our experiments. We also find that theoretical properties
concerning robustness against strategic reporting are somewhat noisy predictors
of empirical performance. Further, there is an apparent trade-off between our
two dimensions of analysis. The best-performing mechanisms in terms of
measurement integrity are highly susceptible to strategic reporting.
Ultimately, however, we show that supplementing mechanisms with realistic
parametric statistical models can, in some cases, improve performance along
both dimensions of our analysis and result in mechanisms that strike the best
balance between them.Comment: The code for our experiments is hosted in the following GitHub
repository:
https://github.com/burrelln/Measurement-Integrity-and-Peer-Assessment.
Version 2 (uploaded on 9/22/22) introduces experiments with real peer grading
data alongside significant changes to the framing of the paper and
presentation of the result
- …