402 research outputs found
Dominantly Truthful Multi-task Peer Prediction with a Constant Number of Tasks
In the setting where participants are asked multiple similar possibly
subjective multi-choice questions (e.g. Do you like Panda Express? Y/N; do you
like Chick-fil-A? Y/N), a series of peer prediction mechanisms are designed to
incentivize honest reports and some of them achieve dominantly truthfulness:
truth-telling is a dominant strategy and strictly dominate other
"non-permutation strategy" with some mild conditions. However, a major issue
hinders the practical usage of those mechanisms: they require the participants
to perform an infinite number of tasks. When the participants perform a finite
number of tasks, these mechanisms only achieve approximated dominant
truthfulness. The existence of a dominantly truthful multi-task peer prediction
mechanism that only requires a finite number of tasks remains to be an open
question that may have a negative result, even with full prior knowledge.
This paper answers this open question by proposing a new mechanism,
Determinant based Mutual Information Mechanism (DMI-Mechanism), that is
dominantly truthful when the number of tasks is at least 2C and the number of
participants is at least 2. C is the number of choices for each question (C=2
for binary-choice questions). In addition to incentivizing honest reports,
DMI-Mechanism can also be transferred into an information evaluation rule that
identifies high-quality information without verification when there are at
least 3 participants. To the best of our knowledge, DMI-Mechanism is the first
dominantly truthful mechanism that works for a finite number of tasks, not to
say a small constant number of tasks.Comment: To appear in SODA2
More Dominantly Truthful Multi-Task Peer Prediction with a Finite Number of Tasks
In the setting where we ask participants multiple similar possibly subjective
multi-choice questions (e.g. Do you like Bulbasaur? Y/N; do you like Squirtle?
Y/N), peer prediction aims to design mechanisms that encourage honest feedback
without verification. A series of works have successfully designed multi-task
peer prediction mechanisms where reporting truthfully is better than any other
strategy (dominantly truthful), while they require an infinite number of tasks.
A recent work proposes the first multi-task peer prediction mechanism,
Determinant Mutual Information (DMI)-Mechanism, where not only is dominantly
truthful but also works for a finite number of tasks (practical). However, the
existence of other practical dominantly-truthful multi-task peer prediction
mechanisms remains to be an open question. This work answers the above question
by providing 1. a new family of information-monotone information measures:
volume mutual information (VMI), where DMI is a special case; 2. a new family
of practical dominantly-truthful multi-task peer prediction mechanisms,
VMI-Mechanisms. To illustrate the importance of VMI-Mechanisms, we also provide
a tractable effort incentive optimization goal. We show that DMI-Mechanism may
not be not optimal but we can construct a sequence of VMI-Mechanisms that are
approximately optimal. The main technical highlight in this paper is a novel
geometric information measure, Volume Mutual Information, that is based on a
simple idea: we can measure an object A's information amount by the number of
objects that is less informative than A. Different densities over the object
lead to different information measures. This also gives Determinant Mutual
Information a simple geometric interpretation
Learning and Strongly Truthful Multi-Task Peer Prediction: A Variational Approach
Peer prediction mechanisms incentivize agents to truthfully report their
signals even in the absence of verification by comparing agents' reports with
those of their peers. In the detail-free multi-task setting, agents respond to
multiple independent and identically distributed tasks, and the mechanism does
not know the prior distribution of agents' signals. The goal is to provide an
-strongly truthful mechanism where truth-telling rewards agents
"strictly" more than any other strategy profile (with additive
error), and to do so while requiring as few tasks as possible. We design a
family of mechanisms with a scoring function that maps a pair of reports to a
score. The mechanism is strongly truthful if the scoring function is "prior
ideal," and -strongly truthful as long as the scoring function is
sufficiently close to the ideal one. This reduces the above mechanism design
problem to a learning problem -- specifically learning an ideal scoring
function. We leverage this reduction to obtain the following three results. 1)
We show how to derive good bounds on the number of tasks required for different
types of priors. Our reduction applies to myriad continuous signal space
settings. This is the first peer-prediction mechanism on continuous signals
designed for the multi-task setting. 2) We show how to turn a soft-predictor of
an agent's signals (given the other agents' signals) into a mechanism. This
allows the practical use of machine learning algorithms that give good results
even when many agents provide noisy information. 3) For finite signal spaces,
we obtain -strongly truthful mechanisms on any stochastically
relevant prior, which is the maximal possible prior. In contrast, prior work
only achieves a weaker notion of truthfulness (informed truthfulness) or
requires stronger assumptions on the prior.Comment: 39 pages, 1 figur
Eliciting and Aggregating Information: An Information Theoretic Approach
Crowdsourcing---outsourcing tasks to a crowd of workers (e.g. Amazon Mechanical Turk, peer grading for massive open online courseware (MOOCs), scholarly peer review, and Yahoo answers)---is a fast, cheap, and effective method for performing simple tasks even at large scales. Two central problems in this area are:
Information Elicitation: how to design reward systems that incentivize high quality feedback from agents; and
Information Aggregation: how to aggregate the collected feedback to obtain a high quality forecast.
This thesis shows that the combination of game theory, information theory, and learning theory can bring a unified framework to both of the central problems in crowdsourcing area. This thesis builds a natural connection between information elicitation and information aggregation, distills the essence of eliciting and aggregating information to the design of proper information measurements and applies the information measurements to both the central problems:
In the setting where information cannot be verified, this thesis proposes a simple yet powerful information theoretical framework, the emph{Mutual Information Paradigm (MIP)}, for information elicitation mechanisms. The framework pays every agent a measure of mutual information between her signal and a peer's signal. The mutual information measurement is required to have the key property that any ``data processing'' on the two random variables will decrease the mutual information between them. We identify such information measures that generalize Shannon mutual information. MIP overcomes the two main challenges in information elicitation without verification: (1) how to incentivize effort and avoid agents colluding to report random or identical responses (2) how to motivate agents who believe they are in the minority to report truthfully.
To elicit expertise without verification, this thesis also defines a natural model for this setting based on the assumption that emph{more sophisticated agents know the beliefs of less sophisticated agents} and extends MIP to a mechanism design framework, the emph{Hierarchical Mutual Information Paradigm (HMIP)}, for this setting.
Aided by the information measures and the frameworks, this thesis (1) designs several novel information elicitation mechanisms (e.g. the disagreement mechanism, the -mutual information mechanism, the multi-hierarchical mutual information mechanism, the common ground mechanism) in various of settings such that honesty and efforts are incentivized and expertise is identified; (2) addresses an important unsupervised learning problem---co-training by reducing it to an information elicitation problem---forecast elicitation without verification.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/145809/1/yuqkong_1.pd
Measurement Integrity in Peer Prediction: A Peer Assessment Case Study
We propose measurement integrity, a property related to ex post reward
fairness, as a novel desideratum for peer prediction mechanisms in many natural
applications. Like robustness against strategic reporting, the property that
has been the primary focus of the peer prediction literature, measurement
integrity is an important consideration for understanding the practical
performance of peer prediction mechanisms. We perform computational
experiments, both with an agent-based model and with real data, to empirically
evaluate peer prediction mechanisms according to both of these important
properties. Our evaluations simulate the application of peer prediction
mechanisms to peer assessment -- a setting in which ex post fairness concerns
are particularly salient. We find that peer prediction mechanisms, as proposed
in the literature, largely fail to demonstrate significant measurement
integrity in our experiments. We also find that theoretical properties
concerning robustness against strategic reporting are somewhat noisy predictors
of empirical performance. Further, there is an apparent trade-off between our
two dimensions of analysis. The best-performing mechanisms in terms of
measurement integrity are highly susceptible to strategic reporting.
Ultimately, however, we show that supplementing mechanisms with realistic
parametric statistical models can, in some cases, improve performance along
both dimensions of our analysis and result in mechanisms that strike the best
balance between them.Comment: The code for our experiments is hosted in the following GitHub
repository:
https://github.com/burrelln/Measurement-Integrity-and-Peer-Assessment.
Version 2 (uploaded on 9/22/22) introduces experiments with real peer grading
data alongside significant changes to the framing of the paper and
presentation of the result
Peer Prediction for Learning Agents
Peer prediction refers to a collection of mechanisms for eliciting
information from human agents when direct verification of the obtained
information is unavailable. They are designed to have a game-theoretic
equilibrium where everyone reveals their private information truthfully. This
result holds under the assumption that agents are Bayesian and they each adopt
a fixed strategy across all tasks. Human agents however are observed in many
domains to exhibit learning behavior in sequential settings. In this paper, we
explore the dynamics of sequential peer prediction mechanisms when participants
are learning agents. We first show that the notion of no regret alone for the
agents' learning algorithms cannot guarantee convergence to the truthful
strategy. We then focus on a family of learning algorithms where strategy
updates only depend on agents' cumulative rewards and prove that agents'
strategies in the popular Correlated Agreement (CA) mechanism converge to
truthful reporting when they use algorithms from this family. This family of
algorithms is not necessarily no-regret, but includes several familiar
no-regret learning algorithms (e.g multiplicative weight update and Follow the
Perturbed Leader) as special cases. Simulation of several algorithms in this
family as well as the -greedy algorithm, which is outside of this
family, shows convergence to the truthful strategy in the CA mechanism.Comment: 34 pages, 9 figure
Optimal Scoring Rules for Multi-dimensional Effort
This paper develops a framework for the design of scoring rules to optimally
incentivize an agent to exert a multi-dimensional effort. This framework is a
generalization to strategic agents of the classical knapsack problem (cf.
Briest, Krysta, and V\"ocking, 2005, Singer, 2010) and it is foundational to
applying algorithmic mechanism design to the classroom. The paper identifies
two simple families of scoring rules that guarantee constant approximations to
the optimal scoring rule. The truncated separate scoring rule is the sum of
single dimensional scoring rules that is truncated to the bounded range of
feasible scores. The threshold scoring rule gives the maximum score if reports
exceed a threshold and zero otherwise. Approximate optimality of one or the
other of these rules is similar to the bundling or selling separately result of
Babaioff, Immorlica, Lucier, and Weinberg (2014). Finally, we show that the
approximate optimality of the best of those two simple scoring rules is robust
when the agent's choice of effort is made sequentially
On Connections Between Machine Learning And Information Elicitation, Choice Modeling, And Theoretical Computer Science
Machine learning, which has its origins at the intersection of computer science and statistics, is now a rapidly growing area of research that is being integrated into almost every discipline in science and business such as economics, marketing and information retrieval. As a consequence of this integration, it is necessary to understand how machine learning interacts with these disciplines and to understand fundamental questions that arise at the resulting interfaces. The goal of my thesis research is to study these interdisciplinary questions at the interface of machine learning and other disciplines including mechanism design/information elicitation, preference/choice modeling, and theoretical computer science
- …