Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets

Bayesian optimization has become a successful tool for hyperparameter optimization of machine learning algorithms, such as support vector machines or deep neural networks. Despite its success, for large datasets, training and validating a single configuration often takes hours, days, or even weeks, which limits the achievable performance. To accelerate hyperparameter optimization, we propose a generative model for the validation error as a function of training set size, which is learned during the optimization process and allows exploration of preliminary configurations on small subsets, by extrapolating to the full dataset. We construct a Bayesian optimization procedure, dubbed Fabolas, which models loss and training time as a function of dataset size and automatically trades off high information gain about the global optimum against computational cost. Experiments optimizing support vector machines and deep neural networks show that Fabolas often finds high-quality solutions 10 to 100 times faster than other state-of-the-art Bayesian optimization methods or the recently proposed bandit strategy Hyperband

Auto-Sklearn 2.0: The Next Generation

Automated Machine Learning, which supports practitioners and researchers with the tedious task of manually designing machine learning pipelines, has recently achieved substantial success. In this paper we introduce new Automated Machine Learning (AutoML) techniques motivated by our winning submission to the second ChaLearn AutoML challenge, PoSH Auto-sklearn. For this, we extend Auto-sklearn with a new, simpler meta-learning technique, improve its way of handling iterative algorithms and enhance it with a successful bandit strategy for budget allocation. Furthermore, we go one step further and study the design space of AutoML itself and propose a solution towards truly hand-free AutoML. Together, these changes give rise to the next generation of our AutoML system, Auto-sklearn (2.0). We verify the improvement by these additions in a large experimental study on 39 AutoML benchmark datasets and conclude the paper by comparing to Auto-sklearn (1.0), reducing the regret by up to a factor of five

Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning

Automated Machine Learning (AutoML) supports practitioners and researchers with the tedious task of designing machine learning pipelines and has recently achieved substantial success. In this paper, we introduce new AutoML approaches motivated by our winning submission to the second ChaLearn AutoML challenge. We develop PoSH Auto-sklearn, which enables AutoML systems to work well on large datasets under rigid time limits by using a new, simple and meta-feature-free meta-learning technique and by employing a successful bandit strategy for budget allocation. However, PoSH Auto-sklearn introduces even more ways of running AutoML and might make it harder for users to set it up correctly. Therefore, we also go one step further and study the design space of AutoML itself, proposing a solution towards truly hands-free AutoML. Together, these changes give rise to the next generation of our AutoML system, Auto-sklearn 2.0. We verify the improvements by these additions in an extensive experimental study on 39 AutoML benchmark datasets. We conclude the paper by comparing to other popular AutoML frameworks and Auto-sklearn 1.0, reducing the relative error by up to a factor of 4.5, and yielding a performance in 10 minutes that is substantially better than what Auto-sklearn 1.0 achieves within an hour

Astraeus - III. The environment and physical properties of reionization sources

In this work, we use the {\sc astraeus} (seminumerical rAdiative tranSfer coupling of galaxy formaTion and Reionization in N-body dArk mattEr simUlationS) framework which couples galaxy formation and reionization in the first billion years. Exploring a number of models for reionization feedback and the escape fraction of ionizing radiation from the galactic environment ($f_\mathrm{esc}$), we quantify how the contribution of star-forming galaxies {(with halo masses $M_h>10^{8.2}$M$_\odot$)} to reionization depends on the radiative feedback model, $f_\mathrm{esc}$, and the environmental over-density. Our key findings are: (i) for constant $f_\mathrm{esc}$ models, intermediate-mass galaxies (with halo masses of $M_h\simeq10^{9-11}$M$_\odot$ and absolute UV magnitudes of $M_{UV} \sim -15$ to $-20$) in intermediate-density regions drive reionization; (ii) scenarios where $f_\mathrm{esc}$ increases with decreasing halo mass shift the galaxy population driving reionization to lower-mass galaxies ($M_h\lesssim10^{9.5}$M$_\odot$) with lower luminosities ($M_{UV} \gtrsim-16$) and over-densities; (iii) reionization imprints its topology on the ionizing emissivity of low-mass galaxies ($M_h\lesssim10^{9}$M$_\odot$) through radiative feedback. Low-mass galaxies experience a stronger suppression of star formation by radiative feedback and show lower ionizing emissivities in over-dense regions; (iv) a change in $f_\mathrm{esc}$ with galaxy properties has the largest impact on the sources of reionization and their detectability, with the radiative feedback strength and environmental over-density playing a sub-dominant role; (v) JWST-surveys (with a limiting magnitude of $M_{UV} = -16$) will be able to detect the galaxies providing $\sim 60-70\%$ ($\sim 10\%$) of reionization photons at $z=7$ for constant $f_\mathrm{esc}$ models (scenarios where $f_\mathrm{esc}$ increases with decreasing halo mass).Comment: 14 pages, 13 figures, accepted for publication in MNRA

A Primer on the Differential Calculus of 3D Orientations

The proper handling of 3D orientations is a central element in many optimization problems in engineering. Unfortunately many researchers and engineers struggle with the formulation of such problems and often fall back to suboptimal solutions. The existence of many different conventions further complicates this issue, especially when interfacing multiple differing implementations. This document discusses an alternative approach which makes use of a more abstract notion of 3D orientations. The relative orientation between two coordinate systems is primarily identified by the coordinate mapping it induces. This is combined with the standard exponential map in order to introduce representation-independent and minimal differentials, which are very convenient in optimization based methods