Multi-Task Bayesian Optimization

Bayesian optimization has recently been proposed as a framework for automatically tuning the hyperparameters of machine learning models and has been shown to yield state-of-the-art performance with impressive ease and efficiency. In this paper, we explore whether it is possible to transfer the knowledge gained from previous optimizations to new tasks in order to find optimal hyperparameter settings more efficiently. Our approach is based on extending multi-task Gaussian processes to the framework of Bayesian optimization. We show that this method significantly speeds up the optimization process when compared to the standard single-task approach. We further propose a straightforward extension of our algorithm in order to jointly minimize the average error across multiple tasks and demonstrate how this can be used to greatly speed up $k$-fold cross-validation. Lastly, our most significant contribution is an adaptation of a recently proposed acquisition function, entropy search, to the cost-sensitive and multi-task settings. We demonstrate the utility of this new acquisition function by utilizing a small dataset in order to explore hyperparameter settings for a large dataset. Our algorithm dynamically chooses which dataset to query in order to yield the most information per unit cost.Engineering and Applied Science

Efficient structure search with multi-task Bayesian optimization

Computational materials science aims to discover new functional materials and optimize their properties, which often includes resource-intensive calculations. To address structure search tasks with the least number of expensive calculations, the Bayesian Optimization Structure Search (BOSS) algorithm has been implemented. BOSS applies active learning in combination with Gaussian process regression to sample-efficiently optimize a target function, which in this case represents the total energy of the material. Materials can be simulated with approximated methods which are fast but less accurate or with costly and accurate electronic structure methods. This work investigates how BOSS can become even more resource-efficient by incorporating calculations from different levels of accuracy. Multi-fidelity BOSS uses the Intrinsic Model of Coregionalization (ICM) to integrate data from different atomistic simulators, all focusing on the same objective, the total energy of the material. This work focuses on multi-fidelity learning acquisition functions, which are one of the key components of the multi-fidelity algorithm. In particular, I developed and implemented several multi-fidelity acquisition functions. To test the functions, I applied multi-fidelity BOSS on the alanine structure search task, where I used simulations of the alanine system based on force fields (AMBER18), density-functional theory (FHI-aims with PBE-exchange correlation functional) and quantum chemistry accuracy (Gaussian16 with CCSD(T)). I found that multi-fidelity BOSS reduced the CPU cost by up to 90% CPU when used with the ELCB or MES acquisition functions. Both acquisition functions enable large savings when used in combination with different separable or inseparable sampling strategies. I also found, that the possible savings depends significantly on the sampling costs of the atomistic simulators, the correlation between the different fidelities and the dimension of the search space

MUMBO:MUlti-task Max-value Bayesian Optimization

We propose MUMBO, the first high-performing yet computationally efficient acquisition function for multi-task Bayesian optimization. Here, the challenge is to perform efficient optimization by evaluating low-cost functions somehow related to our true target function. This is a broad class of problems including the popular task of multi-fidelity optimization. However, while information-theoretic acquisition functions are known to provide state-of-the-art Bayesian optimization, existing implementations for multi-task scenarios have prohibitive computational requirements. Previous acquisition functions have therefore been suitable only for problems with both low-dimensional parameter spaces and function query costs sufficiently large to overshadow very significant optimization overheads. In this work, we derive a novel multi-task version of entropy search, delivering robust performance with low computational overheads across classic optimization challenges and multi-task hyper-parameter tuning. MUMBO is scalable and efficient, allowing multi-task Bayesian optimization to be deployed in problems with rich parameter and fidelity spaces

Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets

Bayesian optimization has become a successful tool for hyperparameter optimization of machine learning algorithms, such as support vector machines or deep neural networks. Despite its success, for large datasets, training and validating a single configuration often takes hours, days, or even weeks, which limits the achievable performance. To accelerate hyperparameter optimization, we propose a generative model for the validation error as a function of training set size, which is learned during the optimization process and allows exploration of preliminary configurations on small subsets, by extrapolating to the full dataset. We construct a Bayesian optimization procedure, dubbed Fabolas, which models loss and training time as a function of dataset size and automatically trades off high information gain about the global optimum against computational cost. Experiments optimizing support vector machines and deep neural networks show that Fabolas often finds high-quality solutions 10 to 100 times faster than other state-of-the-art Bayesian optimization methods or the recently proposed bandit strategy Hyperband