19 research outputs found
Statistical Inference and A/B Testing for First-Price Pacing Equilibria
We initiate the study of statistical inference and A/B testing for
first-price pacing equilibria (FPPE). The FPPE model captures the dynamics
resulting from large-scale first-price auction markets where buyers use
pacing-based budget management. Such markets arise in the context of internet
advertising, where budgets are prevalent.
We propose a statistical framework for the FPPE model, in which a limit FPPE
with a continuum of items models the long-run steady-state behavior of the
auction platform, and an observable FPPE consisting of a finite number of items
provides the data to estimate primitives of the limit FPPE, such as revenue,
Nash social welfare (a fair metric of efficiency), and other parameters of
interest. We develop central limit theorems and asymptotically valid confidence
intervals. Furthermore, we establish the asymptotic local minimax optimality of
our estimators. We then show that the theory can be used for conducting
statistically valid A/B testing on auction platforms. Numerical simulations
verify our central limit theorems, and empirical coverage rates for our
confidence intervals agree with our theory.Comment: - fix referenc
Modeling Interference Using Experiment Roll-out
Experiments on online marketplaces and social networks suffer from
interference, where the outcome of a unit is impacted by the treatment status
of other units. We propose a framework for modeling interference using a
ubiquitous deployment mechanism for experiments, staggered roll-out designs,
which slowly increase the fraction of units exposed to the treatment to
mitigate any unanticipated adverse side effects. Our main idea is to leverage
the temporal variations in treatment assignments introduced by roll-outs to
model the interference structure. We first present a set of model
identification conditions under which the estimation of common estimands is
possible and show how these conditions are aided by roll-out designs. Since
there are often multiple competing models of interference in practice, we then
develop a model selection method that evaluates models based on their ability
to explain outcome variation observed along the roll-out. Through simulations,
we show that our heuristic model selection method, Leave-One-Period-Out,
outperforms other baselines. We conclude with a set of considerations,
robustness checks, and potential limitations for practitioners wishing to use
our framework
Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approach
Suppose an online platform wants to compare a treatment and control policy,
e.g., two different matching algorithms in a ridesharing system, or two
different inventory management algorithms in an online retail site. Standard
randomized controlled trials are typically not feasible, since the goal is to
estimate policy performance on the entire system. Instead, the typical current
practice involves dynamically alternating between the two policies for fixed
lengths of time, and comparing the average performance of each over the
intervals in which they were run as an estimate of the treatment effect.
However, this approach suffers from *temporal interference*: one algorithm
alters the state of the system as seen by the second algorithm, biasing
estimates of the treatment effect. Further, the simple non-adaptive nature of
such designs implies they are not sample efficient.
We develop a benchmark theoretical model in which to study optimal
experimental design for this setting. We view testing the two policies as the
problem of estimating the steady state difference in reward between two unknown
Markov chains (i.e., policies). We assume estimation of the steady state reward
for each chain proceeds via nonparametric maximum likelihood, and search for
consistent (i.e., asymptotically unbiased) experimental designs that are
efficient (i.e., asymptotically minimum variance). Characterizing such designs
is equivalent to a Markov decision problem with a minimum variance objective;
such problems generally do not admit tractable solutions. Remarkably, in our
setting, using a novel application of classical martingale analysis of Markov
chains via Poisson's equation, we characterize efficient designs via a succinct
convex optimization problem. We use this characterization to propose a
consistent, efficient online experimental design that adaptively samples the
two Markov chains
The Economics of Risk
This collection offers an economics-based overview of the various facets of risk. It contains six papers that examine a broad array of research relating to risk. Two papers examine risk management and its application to decision making as well as what researchers have learned over the past few decades in their theoretical investigations of risk. The remaining chapters examine how risk plays out in the particular markets in which it has a significant presence, including casino gambling enterprises, agricultural markets, auctions, and health insurance.https://research.upjohn.org/up_press/1176/thumbnail.jp