Scalarizing Functions in Bayesian Multiobjective Optimization

Scalarizing functions have been widely used to convert a multiobjective optimization problem into a single objective optimization problem. However, their use in solving (computationally) expensive multi- and many-objective optimization problems in Bayesian multiobjective optimization is scarce. Scalarizing functions can play a crucial role on the quality and number of evaluations required when doing the optimization. In this article, we study and review 15 different scalarizing functions in the framework of Bayesian multiobjective optimization and build Gaussian process models (as surrogates, metamodels or emulators) on them. We use expected improvement as infill criterion (or acquisition function) to update the models. In particular, we compare different scalarizing functions and analyze their performance on several benchmark problems with different number of objectives to be optimized. The review and experiments on different functions provide useful insights when using and selecting a scalarizing function when using a Bayesian multiobjective optimization method

Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functions

Quantification of the stationary points and the associated basins of attraction of neural network loss surfaces is an important step towards a better understanding of neural network loss surfaces at large. This work proposes a novel method to visualise basins of attraction together with the associated stationary points via gradient-based random sampling. The proposed technique is used to perform an empirical study of the loss surfaces generated by two different error metrics: quadratic loss and entropic loss. The empirical observations confirm the theoretical hypothesis regarding the nature of neural network attraction basins. Entropic loss is shown to exhibit stronger gradients and fewer stationary points than quadratic loss, indicating that entropic loss has a more searchable landscape. Quadratic loss is shown to be more resilient to overfitting than entropic loss. Both losses are shown to exhibit local minima, but the number of local minima is shown to decrease with an increase in dimensionality. Thus, the proposed visualisation technique successfully captures the local minima properties exhibited by the neural network loss surfaces, and can be used for the purpose of fitness landscape analysis of neural networks.Comment: Preprint submitted to the Neural Networks journa

Tuning optimization algorithms under multiple objective function evaluation budgets

Most sensitivity analysis studies of optimization algorithm control parameters are restricted to a single objective function evaluation (OFE) budget. This restriction is problematic because the optimality of control parameter values is dependent not only on the problem’s fitness landscape, but also on the OFE budget available to explore that landscape. Therefore the OFE budget needs to be taken into consideration when performing control parameter tuning. This article presents a new algorithm (tMOPSO) for tuning the control parameter values of stochastic optimization algorithms under a range of OFE budget constraints. Specifically, for a given problem tMOPSO aims to determine multiple groups of control parameter values, each of which results in optimal performance at a different OFE budget. To achieve this, the control parameter tuning problem is formulated as a multi-objective optimization problem. Additionally, tMOPSO uses a noise-handling strategy and control parameter value assessment procedure, which are specialized for tuning stochastic optimization algorithms. Conducted numerical experiments provide evidence that tMOPSO is effective at tuning under multiple OFE budget constraints.National Research Foundation (NRF) of South Africa.http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=4235hb201

Experimental analysis on the operation of Particle Swarm Optimization

In Particle Swarm Optimization, it has been observed that swarms often stall as opposed to converge. A stall occurs when all of the forward progress that could occur is instead rejected as Failed Exploration. Since the swarms particles are in good regions of the search space with the potential to make more progress, the introduction of perturbations to the pbest positions can lead to significant improvements in the performance of standard Particle Swarm Optimization. The pbest perturbation has been supported by a line search technique that can identify unimodal, globally convex, and non-globally convex search spaces, as well as the approximate size of attraction basin. A deeper analysis of the stall condition reveals that it involves clusters of particles that are performing exploitation, and these clusters are separated by individual particles that are performing exploration. This stall pattern can be identified by a newly developed method that is efficient, accurate, real-time, and search space independent. A more targeted (heterogenous) modification for stall is presented for globally convex search spaces