Shrinkage Estimators in Online Experiments

We develop and analyze empirical Bayes Stein-type estimators for use in the estimation of causal effects in large-scale online experiments. While online experiments are generally thought to be distinguished by their large sample size, we focus on the multiplicity of treatment groups. The typical analysis practice is to use simple differences-in-means (perhaps with covariate adjustment) as if all treatment arms were independent. In this work we develop consistent, small bias, shrinkage estimators for this setting. In addition to achieving lower mean squared error these estimators retain important frequentist properties such as coverage under most reasonable scenarios. Modern sequential methods of experimentation and optimization such as multi-armed bandit optimization (where treatment allocations adapt over time to prior responses) benefit from the use of our shrinkage estimators. Exploration under empirical Bayes focuses more efficiently on near-optimal arms, improving the resulting decisions made under uncertainty. We demonstrate these properties by examining seventeen large-scale experiments conducted on Facebook from April to June 2017

End-to-end Optimization of Multistage Recommender Systems

Complex multistage recommender systems that utilize multiple stages of ranking models and non-model parameters at different stages are used to identify a set of items as output, e.g., advertisements to be delivered by an advertising network. This disclosure describes a framework to optimize the non-model parameters to improve recall, defined based on items delivered as output in comparison to groundtruth items. The optimization can be performed offline, using a simulation that takes as input candidate items and labels of items that are known positives. The optimization can improve the quality of recommendations and can reduce computational cost