794 research outputs found
Frugal Optimization for Cost-related Hyperparameters
The increasing demand for democratizing machine learning algorithms calls for
hyperparameter optimization (HPO) solutions at low cost. Many machine learning
algorithms have hyperparameters which can cause a large variation in the
training cost. But this effect is largely ignored in existing HPO methods,
which are incapable to properly control cost during the optimization process.
To address this problem, we develop a new cost-frugal HPO solution. The core of
our solution is a simple but new randomized direct-search method, for which we
prove a convergence rate of and an
-approximation guarantee on the total cost. We provide
strong empirical results in comparison with state-of-the-art HPO methods on
large AutoML benchmarks.Comment: 29 pages (including supplementary appendix
Selecting Near-Optimal Learners via Incremental Data Allocation
We study a novel machine learning (ML) problem setting of sequentially
allocating small subsets of training data amongst a large set of classifiers.
The goal is to select a classifier that will give near-optimal accuracy when
trained on all data, while also minimizing the cost of misallocated samples.
This is motivated by large modern datasets and ML toolkits with many
combinations of learning algorithms and hyper-parameters. Inspired by the
principle of "optimism under uncertainty," we propose an innovative strategy,
Data Allocation using Upper Bounds (DAUB), which robustly achieves these
objectives across a variety of real-world datasets.
We further develop substantial theoretical support for DAUB in an idealized
setting where the expected accuracy of a classifier trained on samples can
be known exactly. Under these conditions we establish a rigorous sub-linear
bound on the regret of the approach (in terms of misallocated data), as well as
a rigorous bound on suboptimality of the selected classifier. Our accuracy
estimates using real-world datasets only entail mild violations of the
theoretical scenario, suggesting that the practical behavior of DAUB is likely
to approach the idealized behavior.Comment: AAAI-2016: The Thirtieth AAAI Conference on Artificial Intelligenc
Benefits of Monotonicity in Safe Exploration with Gaussian Processes
We consider the problem of sequentially maximising an unknown function over a
set of actions while ensuring that every sampled point has a function value
below a given safety threshold. We model the function using kernel-based and
Gaussian process methods, while differing from previous works in our assumption
that the function is monotonically increasing with respect to a safety
variable. This assumption is motivated by various practical applications such
as adaptive clinical trial design and robotics. Taking inspiration from the
GP-UCB and SafeOpt algorithms, we propose an algorithm, monotone safe UCB
(M-SafeUCB) for this task. We show that M-SafeUCB enjoys theoretical guarantees
in terms of safety, a suitably-defined regret notion, and approximately finding
the entire safe boundary. In addition, we illustrate that the monotonicity
assumption yields significant benefits in terms of both the guarantees obtained
and the algorithmic simplicity. We support our theoretical findings by
performing empirical evaluations on a variety of functions
- β¦