621 research outputs found
Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates
The optimization of algorithm (hyper-)parameters is crucial for achieving
peak performance across a wide range of domains, ranging from deep neural
networks to solvers for hard combinatorial problems. The resulting algorithm
configuration (AC) problem has attracted much attention from the machine
learning community. However, the proper evaluation of new AC procedures is
hindered by two key hurdles. First, AC benchmarks are hard to set up. Second
and even more significantly, they are computationally expensive: a single run
of an AC procedure involves many costly runs of the target algorithm whose
performance is to be optimized in a given AC benchmark scenario. One common
workaround is to optimize cheap-to-evaluate artificial benchmark functions
(e.g., Branin) instead of actual algorithms; however, these have different
properties than realistic AC problems. Here, we propose an alternative
benchmarking approach that is similarly cheap to evaluate but much closer to
the original AC problem: replacing expensive benchmarks by surrogate benchmarks
constructed from AC benchmarks. These surrogate benchmarks approximate the
response surface corresponding to true target algorithm performance using a
regression model, and the original and surrogate benchmark share the same
(hyper-)parameter space. In our experiments, we construct and evaluate
surrogate benchmarks for hyperparameter optimization as well as for AC problems
that involve performance optimization of solvers for hard combinatorial
problems, drawing training data from the runs of existing AC procedures. We
show that our surrogate benchmarks capture overall important characteristics of
the AC scenarios, such as high- and low-performing regions, from which they
were derived, while being much easier to use and orders of magnitude cheaper to
evaluate
On the van der Waerden numbers w(2;3,t)
We present results and conjectures on the van der Waerden numbers w(2;3,t)
and on the new palindromic van der Waerden numbers pdw(2;3,t). We have computed
the new number w(2;3,19) = 349, and we provide lower bounds for 20 <= t <= 39,
where for t <= 30 we conjecture these lower bounds to be exact. The lower
bounds for 24 <= t <= 30 refute the conjecture that w(2;3,t) <= t^2, and we
present an improved conjecture. We also investigate regularities in the good
partitions (certificates) to better understand the lower bounds.
Motivated by such reglarities, we introduce *palindromic van der Waerden
numbers* pdw(k; t_0,...,t_{k-1}), defined as ordinary van der Waerden numbers
w(k; t_0,...,t_{k-1}), however only allowing palindromic solutions (good
partitions), defined as reading the same from both ends. Different from the
situation for ordinary van der Waerden numbers, these "numbers" need actually
to be pairs of numbers. We compute pdw(2;3,t) for 3 <= t <= 27, and we provide
lower bounds, which we conjecture to be exact, for t <= 35.
All computations are based on SAT solving, and we discuss the various
relations between SAT solving and Ramsey theory. Especially we introduce a
novel (open-source) SAT solver, the tawSolver, which performs best on the SAT
instances studied here, and which is actually the original DLL-solver, but with
an efficient implementation and a modern heuristic typical for look-ahead
solvers (applying the theory developed in the SAT handbook article of the
second author).Comment: Second version 25 pages, updates of numerical data, improved
formulations, and extended discussions on SAT. Third version 42 pages, with
SAT solver data (especially for new SAT solver) and improved representation.
Fourth version 47 pages, with updates and added explanation
- …