High-contrast algorithm behavior: Observation, conjecture, and experimental design

After extensive experiments with two algorithms, CPLEX and our implementation of all-integer dual simplex, we observed extreme differences between the two on a set of design automation benchmarks. In many cases one of the two would find an optimal solution within seconds while the other timed out at one hour. We conjecture that this contrast is accounted for by the extent to which the constraint matrix can be made block diagonal via row/column permutations. The actual structure of the matrix without the permutations is not important. Our conjecture is made more precise in two steps: (a) crossing minimization is used on a derived graph to achieve desirable permutations of rows and columns; and (b) the degree of randomness (lack of structure) is measured using diffusion, a measure that approximates what a human perceives as lack of structure. Additional experiments on synthetic instances related to the benchmarks add validity to our conjecture. We observe unexpectedly sharp thresholds where, with only slight variation of our measure, the dominance of the algorithms reverses dramatically. The nature of and explanation for this threshold behavior is left for future research as are many other questions. As far as we are aware the approach taken here is unique and, we hope, will inspire other research of its kind. 1

On SAT Instance Classes and a Method for Reliable Performance Experiments with SAT Solvers

A recent series of experiments with a group of state-of-the-art SAT solvers and several well-defined classes of problem instances reports statistically significant performance variability for the solvers. A systematic analysis of the observed performance data, all openly archived on the Web, reveals distributions which we classify into three broad categories: (1) readily characterized with a simple χ2-test, (2) requiring more in-depth analysis by a statistician, (3) incomplete, due to time-out limit reached by specific solvers. The first category includes two well-known distributions: normal and exponential; we use simple first-order criteria to decide the second category and label the distributions as near-normal, near-exponential and heavy-tail. We expect that good models for some if not most of these may be found with parameters that fit either generalized gamma, Weibull, or Pareto distributions. Our experiments show that most SAT solvers exhibit either normal or exponential distribution of execution time (runtime) on many equivalence classes of problem instances. This finding suggests that the basic mathematical framework for these experiments may well be the same as the one used to test the reliability or lifetime of hardware components such as lightbulbs, A/C units, etc. A batch of N replicated hardware components represents an equivalence class of N proble