5,709 research outputs found
Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates
The optimization of algorithm (hyper-)parameters is crucial for achieving
peak performance across a wide range of domains, ranging from deep neural
networks to solvers for hard combinatorial problems. The resulting algorithm
configuration (AC) problem has attracted much attention from the machine
learning community. However, the proper evaluation of new AC procedures is
hindered by two key hurdles. First, AC benchmarks are hard to set up. Second
and even more significantly, they are computationally expensive: a single run
of an AC procedure involves many costly runs of the target algorithm whose
performance is to be optimized in a given AC benchmark scenario. One common
workaround is to optimize cheap-to-evaluate artificial benchmark functions
(e.g., Branin) instead of actual algorithms; however, these have different
properties than realistic AC problems. Here, we propose an alternative
benchmarking approach that is similarly cheap to evaluate but much closer to
the original AC problem: replacing expensive benchmarks by surrogate benchmarks
constructed from AC benchmarks. These surrogate benchmarks approximate the
response surface corresponding to true target algorithm performance using a
regression model, and the original and surrogate benchmark share the same
(hyper-)parameter space. In our experiments, we construct and evaluate
surrogate benchmarks for hyperparameter optimization as well as for AC problems
that involve performance optimization of solvers for hard combinatorial
problems, drawing training data from the runs of existing AC procedures. We
show that our surrogate benchmarks capture overall important characteristics of
the AC scenarios, such as high- and low-performing regions, from which they
were derived, while being much easier to use and orders of magnitude cheaper to
evaluate
Deterministic Parallel (DP)2LL
Microsoft Research Cambridge, MSR-TR-2011-47Parallel Satisfiability is now recognized as an important research area. The wide deployment of multicore platforms combined with the availability of open and challenging SAT instances are behind this recognition. However, the current parallel SAT solvers suffer from a non-deterministic behavior. This is the consequence of their architectures which rely on weak synchronizing in an attempt to maximize performance. This behavior is a clear downside for practitioners, who are used to both runtime and solution reproducibility. In this paper, we propose the first Deterministic Parallel DPLL engine. It is based on a stateof- the-art parallel portfolio architecture and relies on a controlled synchronizing of the different threads. Our experimental results clearly show that our approach preserves the performance of the parallel portfolio approach while ensuring full reproducibility of the results
ASlib: A Benchmark Library for Algorithm Selection
The task of algorithm selection involves choosing an algorithm from a set of
algorithms on a per-instance basis in order to exploit the varying performance
of algorithms over a set of instances. The algorithm selection problem is
attracting increasing attention from researchers and practitioners in AI. Years
of fruitful applications in a number of domains have resulted in a large amount
of data, but the community lacks a standard format or repository for this data.
This situation makes it difficult to share and compare different approaches
effectively, as is done in other, more established fields. It also
unnecessarily hinders new researchers who want to work in this area. To address
this problem, we introduce a standardized format for representing algorithm
selection scenarios and a repository that contains a growing number of data
sets from the literature. Our format has been designed to be able to express a
wide variety of different scenarios. Demonstrating the breadth and power of our
platform, we describe a set of example experiments that build and evaluate
algorithm selection models through a common interface. The results display the
potential of algorithm selection to achieve significant performance
improvements across a broad range of problems and algorithms.Comment: Accepted to be published in Artificial Intelligence Journa
Deterministic Parallel DPLL
International audienceCurrent parallel SAT solvers su er from a non-deterministic behavior. This is the consequence of their architectures which rely on weak synchronizing in an attempt to maximize performance. This behavior is a clear downside for practitioners, who are used to both runtime and solution reproducibility. In this paper, we propose the rst Deterministic Parallel DPLL engine. Our experimental results clearly show that our approach preserves the performance of the parallel portfolio approach while ensuring full reproducibility of the results
A Time Leap Challenge for SAT Solving
We compare the impact of hardware advancement and algorithm advancement for
SAT solving over the last two decades. In particular, we compare 20-year-old
SAT-solvers on new computer hardware with modern SAT-solvers on 20-year-old
hardware. Our findings show that the progress on the algorithmic side has at
least as much impact as the progress on the hardware side.Comment: Authors' version of a paper which is to appear in the proceedings of
CP'202
Phase transitions in project scheduling.
The analysis of the complexity of combinatorial optimization problems has led to the distinction between problems which are solvable in a polynomially bounded amount of time (classified in P) and problems which are not (classified in NP). This implies that the problems in NP are hard to solve whereas the problems in P are not. However, this analysis is based on worst-case scenarios. The fact that a decision problem is shown to be NP-complete or the fact that an optimization problem is shown to be NP-hard implies that, in the worst case, solving it is very hard. Recent computational results obtained with a well known NP-hard problem, namely the resource-constrained project scheduling problem, indicate that many instances are actually easy to solve. These results are in line with those recently obtained by researchers in the area of artificial intelligence, which show that many NP-complete problemsexhibit so-called phase transitions, resulting in a sudden and dramatic change of computational complexity based on one or more order parameters that are characteristic of the system as a whole. In this paper we provide evidence for the existence of phase transitions in various resource-constrained project scheduling problems. We discuss the use of network complexity measures and resource parameters as potential order parameters. We show that while the network complexity measures seem to reveal continuous easy-hard or hard-easy phase-transitions, the resource parameters exhibit an easy-hard-easy transition behaviour.Networks; Problems; Scheduling; Algorithms;
A Supervisory Control Algorithm Based on Property-Directed Reachability
We present an algorithm for synthesising a controller (supervisor) for a
discrete event system (DES) based on the property-directed reachability (PDR)
model checking algorithm. The discrete event systems framework is useful in
both software, automation and manufacturing, as problems from those domains can
be modelled as discrete supervisory control problems. As a formal framework,
DES is also similar to domains for which the field of formal methods for
computer science has developed techniques and tools. In this paper, we attempt
to marry the two by adapting PDR to the problem of controller synthesis. The
resulting algorithm takes as input a transition system with forbidden states
and uncontrollable transitions, and synthesises a safe and
minimally-restrictive controller, correct-by-design. We also present an
implementation along with experimental results, showing that the algorithm has
potential as a part of the solution to the greater effort of formal supervisory
controller synthesis and verification.Comment: 16 pages; presented at Haifa Verification Conference 2017, the final
publication is available at Springer via
https://doi.org/10.1007/978-3-319-70389-3_
- …