Leveraging covariate adjustments at scale in online A/B testing

Companies offering web services routinely run randomized online experiments to estimate the causal impact associated with the adoption of new features and policies on key performance metrics of interest. These experiments are used to estimate a variety of effects: the increase in click rate due to the repositioning of a banner, the impact on subscription rate as a consequence of a discount or special offer, etc. In these settings, even effects whose sizes are very small can have large downstream impacts. The simple difference in means estimator (Splawa-Neyman et al., 1990) is still the standard estimator of choice for many online A/B testing platforms due to its simplicity. This method, however, can fail to detect small effects, even when the experiment contains thousands or millions of observational units. As a by-product of these experiments, however, large amounts of additional data (covariates) are collected. In this paper, we discuss benefits, costs and risks of allowing experimenters to leverage more complicated estimators that make use of covariates when estimating causal effects of interest. We adapt a recently proposed general-purpose algorithm for the estimation of causal effects with covariates to the setting of online A/B tests. Through this paradigm, we implement several covariate-adjusted causal estimators. We thoroughly evaluate their performance at scale, highlighting benefits and shortcomings of different methods. We show on real experiments how "covariate-adjusted" estimators can (i) lead to more precise quantification of the causal effects of interest and (ii) fix issues related to imbalance across treatment arms - a practical concern often overlooked in the literature. In turn, (iii) these more precise estimates can reduce experimentation time, cutting cost and helping to streamline decision-making processes, allowing for faster adoption of beneficial interventions

Structure in combinatorial optimization and its effect on heuristic performance

2013 Fall.Includes bibliographical references.The goal in combinatorial optimization is to find a good solution among a finite set of solutions. In many combinatorial problems, the set of solutions scales at an exponential or greater rate with the instance size. The maximum boolean satisfiability (MAX-SAT) is one such problem that has many important theoretical and practical applications. Due to the exponential growth of the search space, sufficiently large instances of MAX-SAT are intractable for complete solvers. Incomplete solvers, such as stochastic local search (SLS) algorithms are necessary to find solutions in these cases. Many SLS algorithms for MAX-SAT have been developed on randomly generated benchmarks using a uniform distribution. As a result, SLS algorithms for MAX-SAT perform exceptionally well on uniform random instances. However, instances from real-world applications of MAX-SAT have a structure that is not captured in expectation by uniform random problems. The same SLS algorithms that perform well on uniform instances have a drastic drop in performance on structured instances. To better understand the performance drop on structured instances, we examine three characteristics commonly found in real-world applications of MAX-SAT: a power-law distribution of variables, clause lengths following a power-law distribution, and a community structure similar to that found in small-world models. We find that those instances with a community structure and clause lengths following a power-law distribution have a significantly more rugged search space and larger backbones than uniform random instances. These search space properties make it more difficult for SLS algorithms to find good solutions and in part explains the performance drop on industrial instances. In light of these findings, we examine two ways of improving the performance of SLS algorithms on industrial instances. First, we present a method of tractably computing the average evaluation of solutions in a subspace that we call a hyperplane. These averages can be used to estimate the correct setting of the backbone variables, with as high as 90% accuracy on industrial-like instances. By initializing SLS algorithms with these solutions, the search is able to find significantly better solutions than using standard initialization methods. Second, we re-examine the trade-offs between first and best improving search. We find that in many cases, the evaluation of solutions found by SLS algorithms using first improving search are no worse, and sometimes better, than those found by best improving. First improving search is significantly faster; using first improving search with AdaptG2WSAT, a state-of-the-art SLS algorithm for MAX-SAT, gives us more than a 1,000x speedup on large industrial instances. Finally, we use our hyperplane averages to improve the performance of complete solvers of the satisfiability problem (SAT), the decision version of MAX-SAT. We use the averages to heuristically select a promising hyperplane and perform a reduction of the original problem based on the chosen hyperplane. This significantly reduces the size of the space that must be searched by the complete solver. Using hyperplane reduction as a preprocessing step, a standard complete SAT solver is able to outperform many state-of-the-art complete solvers. Our contributions have advanced the understanding of structured instances and the performance of both SLS algorithms and complete solvers on these instances. We also hope that this work will serve as a foundation for developing better heuristics and complete solvers for real-world applications of SAT and MAX-SAT

Improving Lin-Kernighan-Helsgaun with crossover on clustered instances of the TSP

Abstract. Multi-trial Lin-Kernighan-Helsgaun 2 (LKH-2) is widely considered to be the best Interated Local Search heuristic for the Traveling Salesman Problem (TSP) and has found the best-known solutions to a large number of benchmark problems. Although LKH-2 performs exceptionally well on most instances, it is known to have difficulty on clustered instances of the TSP. Generalized Partition Crossover (GPX) is a crossover operator for the TSP that efficiently constructs new solutions by partitioning a graph constructed from the union of two solutions. We show that GPX is especially well-suited for clustered instances and evaluate its ability to improve solutions found by LKH-2. We present two methods of combining GPX with multi-trial LKH-2. We find that combining GPX with LKH-2 dramatically improves the evaluation of solutions found by LKH-2 alone on clustered instances with sizes ranging from 3,000 to 30,000 cities

Greedy or Not? Best Improving versus First Improving Stochastic Local Search for MAXSAT

Stochastic local search (SLS) is the dominant paradigm for incomplete SAT and MAXSAT solvers. Early studies on small 3SAT instances found that the use of “best improving” moves did not improve search compared to using an arbitrary “first improving ” move. Yet SLS algorithms continue to use best improving moves. We revisit this issue by studying very large random and industrial MAXSAT problems. Because locating best improving moves is more expensive than first improving moves, we designed an “approximate best ” improving move algorithm and prove that it is as efficient as first improving move SLS. For industrial problems the first local optima found using best improving moves are statistically significantly better than local optima found using first improving moves. However, this advantage reverses as search continues and algorithms must explore equal moves on plateaus. This reversal appears to be associated with critical variables that are in many clauses and that also yield large improving moves. Local Search for MAXSAT Stochastic local search algorithms have successfully solved large random instances as well as industrial instances of both SAT and MAXSAT. “Best improving ” local search for MAXSAT identifies all of the improving moves in the Hamming distance 1 neighborhood, then uses the best improving move to update the incumbent solution. In contrast, “first improving ” local search arbitrarily selects the first available improving move to update the incumbent solution. Many stochastic local search (SLS) algorithms for MAXSAT can be viewed as having two stages: a greedy stage that uses best improving move local search followed by an exploratory stage that heuristically guides search after plateaus (large regions in the search space in which the majority of moves have the same evaluation) have been encountere

Hyperplane Initialized Local Search for MAXSAT

By converting the MAXSAT problem to Walsh polynomials, we can efficiently and exactly compute the hyperplane averages of fixed order k. We use this fact to construct initial solutions based on variable configurations that maximize the sampling of hyperplanes with good average evaluations. The Walsh coefficients can also be used to implement a constant time neighborhood update which is integral to a fast next descent local search for MAXSAT (and for all bounded pseudo-Boolean optimization problems.) We evaluate the effect of initializing local search with hyperplane averages on both the first local optima found by the search and the final solutions found after a fixed number of bit flips. Hyperplane initialization not only provides better evaluations, but also finds local optima closer to the globally optimal solution in fewer bit flips than search initialized with random solutions. A next descent search initialized with hyperplane averages is able to outperform several state-of-the art stochastic local search algorithms on both random and industrial instances of MAXSAT