14 research outputs found
Warmstarting of Model-based Algorithm Configuration
The performance of many hard combinatorial problem solvers depends strongly
on their parameter settings, and since manual parameter tuning is both tedious
and suboptimal the AI community has recently developed several algorithm
configuration (AC) methods to automatically address this problem. While all
existing AC methods start the configuration process of an algorithm A from
scratch for each new type of benchmark instances, here we propose to exploit
information about A's performance on previous benchmarks in order to warmstart
its configuration on new types of benchmarks. We introduce two complementary
ways in which we can exploit this information to warmstart AC methods based on
a predictive model. Experiments for optimizing a very flexible modern SAT
solver on twelve different instance sets show that our methods often yield
substantial speedups over existing AC methods (up to 165-fold) and can also
find substantially better configurations given the same compute budget.Comment: Preprint of AAAI'18 pape
Learning Multiple Defaults for Machine Learning Algorithms
The performance of modern machine learning methods highly depends on their
hyperparameter configurations. One simple way of selecting a configuration is
to use default settings, often proposed along with the publication and
implementation of a new algorithm. Those default values are usually chosen in
an ad-hoc manner to work good enough on a wide variety of datasets. To address
this problem, different automatic hyperparameter configuration algorithms have
been proposed, which select an optimal configuration per dataset. This
principled approach usually improves performance, but adds additional
algorithmic complexity and computational costs to the training procedure. As an
alternative to this, we propose learning a set of complementary default values
from a large database of prior empirical results. Selecting an appropriate
configuration on a new dataset then requires only a simple, efficient and
embarrassingly parallel search over this set. We demonstrate the effectiveness
and efficiency of the approach we propose in comparison to random search and
Bayesian Optimization
OBOE: Collaborative Filtering for AutoML Model Selection
Algorithm selection and hyperparameter tuning remain two of the most
challenging tasks in machine learning. Automated machine learning (AutoML)
seeks to automate these tasks to enable widespread use of machine learning by
non-experts. This paper introduces OBOE, a collaborative filtering method for
time-constrained model selection and hyperparameter tuning. OBOE forms a matrix
of the cross-validated errors of a large number of supervised learning models
(algorithms together with hyperparameters) on a large number of datasets, and
fits a low rank model to learn the low-dimensional feature vectors for the
models and datasets that best predict the cross-validated errors. To find
promising models for a new dataset, OBOE runs a set of fast but informative
algorithms on the new dataset and uses their cross-validated errors to infer
the feature vector for the new dataset. OBOE can find good models under
constraints on the number of models fit or the total time budget. To this end,
this paper develops a new heuristic for active learning in time-constrained
matrix completion based on optimal experiment design. Our experiments
demonstrate that OBOE delivers state-of-the-art performance faster than
competing approaches on a test bed of supervised learning problems. Moreover,
the success of the bilinear model used by OBOE suggests that AutoML may be
simpler than was previously understood
Auto-Sklearn 2.0: The Next Generation
Automated Machine Learning, which supports practitioners and researchers with
the tedious task of manually designing machine learning pipelines, has recently
achieved substantial success. In this paper we introduce new Automated Machine
Learning (AutoML) techniques motivated by our winning submission to the second
ChaLearn AutoML challenge, PoSH Auto-sklearn. For this, we extend Auto-sklearn
with a new, simpler meta-learning technique, improve its way of handling
iterative algorithms and enhance it with a successful bandit strategy for
budget allocation. Furthermore, we go one step further and study the design
space of AutoML itself and propose a solution towards truly hand-free AutoML.
Together, these changes give rise to the next generation of our AutoML system,
Auto-sklearn (2.0). We verify the improvement by these additions in a large
experimental study on 39 AutoML benchmark datasets and conclude the paper by
comparing to Auto-sklearn (1.0), reducing the regret by up to a factor of five