Online Evaluation for Effective Web Service Development

Development of the majority of the leading web services and software products today is generally guided by data-driven decisions based on evaluation that ensures a steady stream of updates, both in terms of quality and quantity. Large internet companies use online evaluation on a day-to-day basis and at a large scale. The number of smaller companies using A/B testing in their development cycle is also growing. Web development across the board strongly depends on quality of experimentation platforms. In this tutorial, we overview state-of-the-art methods underlying everyday evaluation pipelines at some of the leading Internet companies. Software engineers, designers, analysts, service or product managers --- beginners, advanced specialists, and researchers --- can learn how to make web service development data-driven and do it effectively

Finite-State Extreme Effect Variable

We generalize to the finite-state case the notion of the extreme effect variable $Y$ that accumulates all the effect of a variant variable $V$ observed in changes of another variable $X$. We conduct theoretical analysis and turn the problem of finding of an effect variable into a problem of a simultaneous decomposition of a set of distributions. The states of the extreme effect variable, on the one hand, are minimally affected by the variant variable $V$ and, on the other hand, are extremely different with respect to the observable variable $X$. We apply our technique to online evaluation of a web search engine through A/B testing and show its utility.Comment: 15 pages; 2 figures; 1 tabl