Bayesian optimization (BO) is a widely popular approach for the
hyperparameter optimization (HPO) of machine learning algorithms. At its core,
BO iteratively evaluates promising configurations until a user-defined budget,
such as wall-clock time or number of iterations, is exhausted. While the final
performance after tuning heavily depends on the provided budget, it is hard to
pre-specify an optimal value in advance. In this work, we propose an effective
and intuitive termination criterion for BO that automatically stops the
procedure if it is sufficiently close to the global optima. Across an extensive
range of real-world HPO problems, we show that our termination criterion
achieves better test performance compared to existing baselines from the
literature, such as stopping when the probability of improvement drops below a
fixed threshold. We also provide evidence that these baselines are, compared to
our method, highly sensitive to the choices of their own hyperparameters.
Additionally, we find that overfitting might occur in the context of HPO, which
is arguably an overlooked problem in the literature, and show that our
termination criterion mitigates this phenomenon on both small and large
datasets.Comment: Submitted to ICLR 202