139 research outputs found
Are we Forgetting about Compositional Optimisers in Bayesian Optimisation?
Bayesian optimisation presents a sample-efficient methodology for global optimisation. Within this framework, a crucial performance-determining subroutine is the maximisation of the acquisition function, a task complicated by the fact that acquisition functions tend to be non-convex and thus nontrivial to optimise. In this paper, we undertake a comprehensive empirical study of approaches to maximise the acquisition function. Additionally, by deriving novel, yet mathematically equivalent, compositional forms for popular acquisition functions, we recast the maximisation task as a compositional optimisation problem, allowing us to benefit from the extensive literature in this field. We highlight the empirical advantages of the compositional approach to acquisition function maximisation across 3958 individual experiments comprising synthetic optimisation tasks as well as tasks from Bayesmark. Given the generality of the acquisition function maximisation subroutine, we posit that the adoption of compositional optimisers has the potential to yield performance improvements across all domains in which Bayesian optimisation is currently being applied. An open-source implementation is made available at https://github.com/huawei-noah/noah-research/tree/CompBO/BO/HEBO/CompBO
Meta-surrogate benchmarking for hyperparameter optimization
Despite the recent progress in hyperparameter optimization (HPO), available
benchmarks that resemble real-world scenarios consist of a few and very large
problem instances that are expensive to solve. This blocks researchers and
practitioners not only from systematically running large-scale comparisons that
are needed to draw statistically significant results but also from reproducing
experiments that were conducted before. This work proposes a method to
alleviate these issues by means of a meta-surrogate model for HPO tasks trained
on off-line generated data. The model combines a probabilistic encoder with a
multi-task model such that it can generate inexpensive and realistic tasks of
the class of problems of interest. We demonstrate that benchmarking HPO methods
on samples of the generative model allows us to draw more coherent and
statistically significant conclusions that can be reached orders of magnitude
faster than using the original tasks. We provide evidence of our findings for
various HPO methods on a wide class of problems
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
Bayesian optimization has become a successful tool for hyperparameter
optimization of machine learning algorithms, such as support vector machines or
deep neural networks. Despite its success, for large datasets, training and
validating a single configuration often takes hours, days, or even weeks, which
limits the achievable performance. To accelerate hyperparameter optimization,
we propose a generative model for the validation error as a function of
training set size, which is learned during the optimization process and allows
exploration of preliminary configurations on small subsets, by extrapolating to
the full dataset. We construct a Bayesian optimization procedure, dubbed
Fabolas, which models loss and training time as a function of dataset size and
automatically trades off high information gain about the global optimum against
computational cost. Experiments optimizing support vector machines and deep
neural networks show that Fabolas often finds high-quality solutions 10 to 100
times faster than other state-of-the-art Bayesian optimization methods or the
recently proposed bandit strategy Hyperband
Efficient tuning in supervised machine learning
The tuning of learning algorithm parameters has become more and more important during the last years. With the fast growth of computational power and available memory databases have grown dramatically. This is very challenging for the tuning of parameters arising in machine learning, since the training can become very time-consuming for large datasets. For this reason efficient tuning methods are required, which are able to improve the predictions of the learning algorithms. In this thesis we incorporate model-assisted optimization techniques, for performing efficient optimization on noisy datasets with very limited budgets. Under this umbrella we also combine learning algorithms with methods for feature construction and selection. We propose to integrate a variety of elements into the learning process. E.g., can tuning be helpful in learning tasks like time series regression using state-of-the-art machine learning algorithms? Are statistical methods capable to reduce noise e ffects? Can surrogate models like Kriging learn a reasonable mapping of the parameter landscape to the quality measures, or are they deteriorated by disturbing factors? Summarizing all these parts, we analyze if superior learning algorithms can be created, with a special focus on efficient runtimes. Besides the advantages of systematic tuning approaches, we also highlight possible obstacles and issues of tuning. Di fferent tuning methods are compared and the impact of their features are exposed. It is a goal of this work to give users insights into applying state-of-the-art learning algorithms profitably in practiceBundesministerium f ür Bildung und Forschung (Germany), Cologne University of Applied Sciences (Germany), Kind-Steinm uller-Stiftung (Gummersbach, Germany)Algorithms and the Foundations of Software technolog
Automatic machine learning:methods, systems, challenges
This open access book presents the first comprehensive overview of general methods in Automatic Machine Learning (AutoML), collects descriptions of existing systems based on these methods, and discusses the first international challenge of AutoML systems. The book serves as a point of entry into this quickly-developing field for researchers and advanced students alike, as well as providing a reference for practitioners aiming to use AutoML in their work. The recent success of commercial ML applications and the rapid growth of the field has created a high demand for off-the-shelf ML methods that can be used easily and without expert knowledge. Many of the recent machine learning successes crucially rely on human experts, who select appropriate ML architectures (deep learning architectures or more traditional ML workflows) and their hyperparameters; however the field of AutoML targets a progressive automation of machine learning, based on principles from optimization and machine learning itself
Using Sequential Statistical Tests for Efficient Hyperparameter Tuning
Hyperparameter tuning is one of the the most time-consuming parts in machine
learning. Despite the existence of modern optimization algorithms that minimize
the number of evaluations needed, evaluations of a single setting may still be
expensive. Usually a resampling technique is used, where the machine learning
method has to be fitted a fixed number of k times on different training
datasets. The respective mean performance of the k fits is then used as
performance estimator. Many hyperparameter settings could be discarded after
less than k resampling iterations if they are clearly inferior to
high-performing settings. However, resampling is often performed until the very
end, wasting a lot of computational effort. To this end, we propose the
Sequential Random Search (SQRS) which extends the regular random search
algorithm by a sequential testing procedure aimed at detecting and eliminating
inferior parameter configurations early. We compared our SQRS with regular
random search using multiple publicly available regression and classification
datasets. Our simulation study showed that the SQRS is able to find similarly
well-performing parameter settings while requiring noticeably fewer
evaluations. Our results underscore the potential for integrating sequential
tests into hyperparameter tuning
- …