2,155 research outputs found
Parallel surrogate-assisted global optimization with expensive functions – a survey
Surrogate assisted global optimization is gaining popularity. Similarly, modern advances in computing power increasingly rely on parallelization rather than faster processors. This paper examines some of the methods used to take advantage of parallelization in surrogate based global optimization. A key issue focused on in this review is how different algorithms balance exploration and exploitation. Most of the papers surveyed are adaptive samplers that employ Gaussian Process or Kriging surrogates. These allow sophisticated approaches for balancing exploration and exploitation and even allow to develop algorithms with calculable rate of convergence as function of the number of parallel processors. In addition to optimization based on adaptive sampling, surrogate assisted parallel evolutionary algorithms are also surveyed. Beyond a review of the present state of the art, the paper also argues that methods that provide easy parallelization, like multiple parallel runs, or methods that rely on population of designs for diversity deserve more attention.United States. Dept. of Energy (National Nuclear Security Administration. Advanced Simulation and Computing Program. Cooperative Agreement under the Predictive Academic Alliance Program. DE-NA0002378
SHADHO: Massively Scalable Hardware-Aware Distributed Hyperparameter Optimization
Computer vision is experiencing an AI renaissance, in which machine learning
models are expediting important breakthroughs in academic research and
commercial applications. Effectively training these models, however, is not
trivial due in part to hyperparameters: user-configured values that control a
model's ability to learn from data. Existing hyperparameter optimization
methods are highly parallel but make no effort to balance the search across
heterogeneous hardware or to prioritize searching high-impact spaces. In this
paper, we introduce a framework for massively Scalable Hardware-Aware
Distributed Hyperparameter Optimization (SHADHO). Our framework calculates the
relative complexity of each search space and monitors performance on the
learning task over all trials. These metrics are then used as heuristics to
assign hyperparameters to distributed workers based on their hardware. We first
demonstrate that our framework achieves double the throughput of a standard
distributed hyperparameter optimization framework by optimizing SVM for MNIST
using 150 distributed workers. We then conduct model search with SHADHO over
the course of one week using 74 GPUs across two compute clusters to optimize
U-Net for a cell segmentation task, discovering 515 models that achieve a lower
validation loss than standard U-Net.Comment: 10 pages, 6 figure
Quantifying uncertainty with ensembles of surrogates for blackbox optimization
This work is in the context of blackbox optimization where the functions
defining the problem are expensive to evaluate and where no derivatives are
available. A tried and tested technique is to build surrogates of the objective
and the constraints in order to conduct the optimization at a cheaper
computational cost. This work proposes different uncertainty measures when
using ensembles of surrogates. The resulting combination of an ensemble of
surrogates with our measures behaves as a stochastic model and allows the use
of efficient Bayesian optimization tools. The method is incorporated in the
search step of the mesh adaptive direct search (MADS) algorithm to improve the
exploration of the search space. Computational experiments are conducted on
seven analytical problems, two multi-disciplinary optimization problems and two
simulation problems. The results show that the proposed approach solves
expensive simulation-based problems at a greater precision and with a lower
computational effort than stochastic models.Comment: 36 pages, 11 figures, submitte
Scalarizing Functions in Bayesian Multiobjective Optimization
Scalarizing functions have been widely used to convert a multiobjective
optimization problem into a single objective optimization problem. However,
their use in solving (computationally) expensive multi- and many-objective
optimization problems in Bayesian multiobjective optimization is scarce.
Scalarizing functions can play a crucial role on the quality and number of
evaluations required when doing the optimization. In this article, we study and
review 15 different scalarizing functions in the framework of Bayesian
multiobjective optimization and build Gaussian process models (as surrogates,
metamodels or emulators) on them. We use expected improvement as infill
criterion (or acquisition function) to update the models. In particular, we
compare different scalarizing functions and analyze their performance on
several benchmark problems with different number of objectives to be optimized.
The review and experiments on different functions provide useful insights when
using and selecting a scalarizing function when using a Bayesian multiobjective
optimization method
Federated Zeroth-Order Optimization using Trajectory-Informed Surrogate Gradients
Federated optimization, an emerging paradigm which finds wide real-world
applications such as federated learning, enables multiple clients (e.g., edge
devices) to collaboratively optimize a global function. The clients do not
share their local datasets and typically only share their local gradients.
However, the gradient information is not available in many applications of
federated optimization, which hence gives rise to the paradigm of federated
zeroth-order optimization (ZOO). Existing federated ZOO algorithms suffer from
the limitations of query and communication inefficiency, which can be
attributed to (a) their reliance on a substantial number of function queries
for gradient estimation and (b) the significant disparity between their
realized local updates and the intended global updates. To this end, we (a)
introduce trajectory-informed gradient surrogates which is able to use the
history of function queries during optimization for accurate and
query-efficient gradient estimation, and (b) develop the technique of adaptive
gradient correction using these gradient surrogates to mitigate the
aforementioned disparity. Based on these, we propose the federated zeroth-order
optimization using trajectory-informed surrogate gradients (FZooS) algorithm
for query- and communication-efficient federated ZOO. Our FZooS achieves
theoretical improvements over the existing approaches, which is supported by
our real-world experiments such as federated black-box adversarial attack and
federated non-differentiable metric optimization
- …