Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates

The optimization of algorithm (hyper-)parameters is crucial for achieving peak performance across a wide range of domains, ranging from deep neural networks to solvers for hard combinatorial problems. The resulting algorithm configuration (AC) problem has attracted much attention from the machine learning community. However, the proper evaluation of new AC procedures is hindered by two key hurdles. First, AC benchmarks are hard to set up. Second and even more significantly, they are computationally expensive: a single run of an AC procedure involves many costly runs of the target algorithm whose performance is to be optimized in a given AC benchmark scenario. One common workaround is to optimize cheap-to-evaluate artificial benchmark functions (e.g., Branin) instead of actual algorithms; however, these have different properties than realistic AC problems. Here, we propose an alternative benchmarking approach that is similarly cheap to evaluate but much closer to the original AC problem: replacing expensive benchmarks by surrogate benchmarks constructed from AC benchmarks. These surrogate benchmarks approximate the response surface corresponding to true target algorithm performance using a regression model, and the original and surrogate benchmark share the same (hyper-)parameter space. In our experiments, we construct and evaluate surrogate benchmarks for hyperparameter optimization as well as for AC problems that involve performance optimization of solvers for hard combinatorial problems, drawing training data from the runs of existing AC procedures. We show that our surrogate benchmarks capture overall important characteristics of the AC scenarios, such as high- and low-performing regions, from which they were derived, while being much easier to use and orders of magnitude cheaper to evaluate

Algorithms for the workflow satisfiability problem engineered for counting constraints

The workflow satisfiability problem (WSP) asks whether there exists an assignment of authorized users to the steps in a workflow specification that satisfies the constraints in the specification. The problem is NP-hard in general, but several subclasses of the problem are known to be fixed-parameter tractable (FPT) when parameterized by the number of steps in the specification. In this paper, we consider the WSP with user-independent counting constraints, a large class of constraints for which the WSP is known to be FPT. We describe an efficient implementation of an FPT algorithm for solving this subclass of the WSP and an experimental evaluation of this algorithm. The algorithm iteratively generates all equivalence classes of possible partial solutions until, whenever possible, it finds a complete solution to the problem. We also provide a reduction from a WSP instance to a pseudo-Boolean SAT instance. We apply this reduction to the instances used in our experiments and solve the resulting PB SAT problems using SAT4J, a PB SAT solver. We compare the performance of our algorithm with that of SAT4J and discuss which of the two approaches would be more effective in practice

A Casual Tour Around a Circuit Complexity Bound

I will discuss the recent proof that the complexity class NEXP (nondeterministic exponential time) lacks nonuniform ACC circuits of polynomial size. The proof will be described from the perspective of someone trying to discover it.Comment: 21 pages, 2 figures. An earlier version appeared in SIGACT News, September 201

Learning Max-CSPs via Active Constraint Acquisition

Constraint acquisition can assist non-expert users to model their problems as constraint networks. In active constraint acquisition, this is achieved through an interaction between the learner, who posts examples, and the user who classifies them as solutions or not. Although there has been recent progress in active constraint acquisition, the focus has only been on learning satisfaction problems with hard constraints. In this paper, we deal with the problem of learning soft constraints in optimization problems via active constraint acquisition, specifically in the context of the Max-CSP. Towards this, we first introduce a new type of queries in the context of constraint acquisition, namely partial preference queries, and then we present a novel algorithm for learning soft constraints in Max-CSPs, using such queries. We also give some experimental results