The rise of internet-based services and products in the late 1990's brought
about an unprecedented opportunity for online businesses to engage in large
scale data-driven decision making. Over the past two decades, organizations
such as Airbnb, Alibaba, Amazon, Baidu, Booking, Alphabet's Google, LinkedIn,
Lyft, Meta's Facebook, Microsoft, Netflix, Twitter, Uber, and Yandex have
invested tremendous resources in online controlled experiments (OCEs) to assess
the impact of innovation on their customers and businesses. Running OCEs at
scale has presented a host of challenges requiring solutions from many domains.
In this paper we review challenges that require new statistical methodologies
to address them. In particular, we discuss the practice and culture of online
experimentation, as well as its statistics literature, placing the current
methodologies within their relevant statistical lineages and providing
illustrative examples of OCE applications. Our goal is to raise academic
statisticians' awareness of these new research opportunities to increase
collaboration between academia and the online industry