The paper focuses on the advantages and drawbacks of different strategies which may be used to assist kriging surrogate modelling with the purpose of selecting multiple design vectors for evaluation when stepping forward in optimisation routines. The combined criteria include the efficiency of finding the global optimum but also the quality of the approximation of the shape of the objective function; the latter may be used to make judgements about the robustness of the optimised design

Rotaru, M.

Sykulski, J.K.

Xiao, Song

English

Southampton (e-Prints Soton)

STRATEGIES FOR BALANCING EXPLORATION AND EXPLOITATION  IN ELECTROMAGNETIC OPTIMISATION    Song Xiao, Mihai Rotaru and Jan K. Sykulski  Electronics and Computer Science (ECS), University of Southampton, Southampton, UK sx1g10@ecs.soton.ac.uk, mr@ecs.soton.ac.uk, jks@soton.ac.uk   Abstract – The paper focuses on the advantages and drawbacks of different strategies which may be used to assist kriging surrogate modelling with the purpose of selecting multiple design vectors for evaluation when stepping forward in optimisation routines. The combined criteria include the efficiency of finding the  global optimum but also the quality of the approximation of the shape of the objective function; the latter may be used to make judgements about the robustness of the optimised design.  I. INTRODUCTION  Design problems in electromagnetic devices are commonly solved using time-consuming numerical techniques, such as the finite element method. In order to relieve the  heavy burden of computation in such  designs,  kriging  has  been suggested as one of the reliable surrogate models with low computational cost and good accuracy of predicting the shape of the objective function. In the optimisation task, the main target is using as few ‘expensive’ objective function calls as possible to find the global optimum. The balance between exploration (searching the region with high uncertainty) and exploitation (searching the highly confident space) has been discussed before  [1-4]. This paper puts main emphasis on improving the existing strategies to predict the shape of the objective function as accurately as possible – in addition to locating  the global optimum –  in order to assess the robustness of the solution.  II. KRIGING WITH DIFFERENT STRATEGIES  A .Kriging and different strategies   Kriging [1]  can exploit the spatial correlation of data in order to predict the shape of the objective function based only on limited information and estimates the accuracy of this prediction, which is helpful in assisting the main decision of the optimisation process how to choose the next design vector for evaluation. In general, an estimate of the accuracy (called the potential error) by the kriging model is commonly used to build a range of different ‘utility functions’  such as the  Expected Improvement  Function (EI)  [2],  or  Weighted Expected Improvement (WEI) [3]. The EI function is defined as If   () 0 sx>  min minminˆˆ () ()ˆ ( () ) ()() () () ()f yx f yxf yx sxsEsIxxφ−−−Φ = +     (1) If   () 0 sx>  0 EI =                                                                                   (2) where  ˆ() yx is the predicted value of objective function by the kriging model, and s(x) is the root mean squared error in this prediction. The first term called the Gaussian density favours searching promising regions, whereas the second term (Gaussian distribution function) is related to exploration, which favours searching regions with high uncertainty. Finding the global optimum of objective function is one of the significant aims for an optimization problem. In practical experiments, the exploration term performs dramatically better in terms of finding the global optimum of the objective function, while the exploitation often can only find the local minimum. Since EI applies equal weights on the two terms, it may be seen as a fixed compromise between exploration and exploitation. The WEI is derived from EI by adding a tuneable parameter which can adjust the weights on exploration and exploitation. As suggested by previous tests [4], the optimal choice of the weights is known as critical in terms of the ability of the algorithm to achieve global optimum and doing it efficiently; unfortunately the optimal weights are normally hard to find and require numerous tests. Therefore two novel algorithms using reinforcement learning [5] called Adaptive Weighted Expected Improvement (AWEI) and Surrogate Model based Weighted Expected Improvement approach with rewards [6] (SMWEI) [4] have been proposed to make the process of tuning weights more intelligent and self-guiding. The Mean Square Error (MSE) from the kriging model is used to calculate the rewards. The AWEI can tune the weights automatically based on the comparison between the potential rewards  from  two different weight  distributions emphasising exploitation and exploration,  respectively.  After comparison, the weights are redistributed on the two terms  of  (1)  to encourage exploration or exploitation depending of the results of the initial pre-test. However, the AWEI only takes account of the short term rewards at a given iteration step, whereas the SMWEI can predict the cumulative rewards likely to occur in long term as a consequence of a particular choice of actions. Furthermore, the SMWEI creates a surrogate model based on potential error and kriging prediction to use in a pre-test rather than using the information from the time-consuming finite element modelling software. In  the  pre-test, two distinct weights are used – one favouring exploration and the other one exploitation – and iterations continue using the surrogate model independently in parallel until overall rewards have been found. The optional weight with better reward of the two is then used to feed back – via the FEM module – into the main iterative loop of the design process.  B. The SMWEI with Multi-weights in pre-test       In practical electromagnetic problems, the robustness of the design is a significant requirement that needs to be considered. Through testing it has been found that the SMWEI algorithm with certain pairs of weights in the pre-test performs better in terms of estimating  the shape of the objective function,  a PNC E2012XXII SymposiumElectromagnetic Phenomena in Nonlinear CircuitsJune 26 - June 29, 2012, Pula, CROATIA1feature which might be helpful when assessing the robustness of the design. As SMWEI is limited by the pre-set pair of weights in the  pre-test,  a number of experiments may be necessary to find the pair resulting in more faithful representation of the shape. As the pre-test is ‘cheap’, more weights can be selected to broaden the base for comparisons. The new version of SMWEI with multi-weights is described in Fig. 1. In the pre-tests, if one of the rewards is not assessed properly or fails, the remaining rewards still participate in the comparison until an action with the biggest reward is chosen.   Fig. 1. The flowchart for SMWEI with multi weights in pre-test  C. The SMWEI with the strategy of adaptively tuning weights   In the pre-test of SMWEI, a pair of fixed weights (one emphasising exploration and the other one exploitation) needs to be set initially. The guidelines how to select such weights are subject to further experiments. However, the strategy of tuning the weights automatically and adaptively in the  pre-test of AWEI can also be used in SMWEI in order to avoid the need for setting initial optional weights. The decision-making chart of the actual implementation is shown in Fig. 1.  Because all pre-tests in SMWEI apply ‘cheap’ simplified surrogate model based on the specific prediction and potential error produced by kriging, the Mean Square Error might be directly used in each pre-test’s remaining iterations instead of the Expected Improvement. The simplified surrogate model in the pre-test, quite rough initially, is increasingly accurate as a result of  adding objective function calls; therefore  the MSE might guide the kriging model directly to search the region of the simplified surrogate model with high uncertainty. Ш. RESULTS  Fig. 2(a) shows one of the test results to approximate the two variables Schwefel function [7], which is a two-objective task. The global minimum has been found after 9 iterations, but the quality of the shape representation of the objective function is poor in Fig. 2(b). Complete results will be reported in the full version.  Fig. 2: a) Two variables Schwefel function in the xϵ[-500 500] and yϵ[-500 500] domain. b) The approximation found by kriging with EI.  IV. CONCLUSIONS  It is argued that a multi reward scheme based on a surrogate model may provide the best prediction of long term benefits for achieving best balance between exploration and exploitation.  REFERENCES  [1]  L. Lebensztajn, C.A.R. Marreto, M.C. Costa, J-L. Coulomb, “Kriging: a useful tool for electromagnetic device optimization”, IEEE Transactions on Magnetics, vol. 40, No. 2, pp. 1196-99, 2004. [2]  D.R. Jones, M. Schonlau, W.J. Welch, “Efficient global optimization of expensive black-box functions”, Journal of Global Optimization, vol. 13, pp. 455-492, 1998. [3]  A.  Sobester,  S.J.  Leary, A.J. Keane,  “On the design of optimization strategies based on global response surface approximation models”, Journal of Global Optimization, vol. 33, pp. 31-59, 2005. [4]  Song Xiao, M. Rotaru, J.K. Sykulski, “Exploration versus exploitation using kriging surrogate modelling in electromagnetic design”, accepted to COMPEL, International journal for computation and mathematics in electrical and electronic engineering, vol. 31, no. 5, 2012. [5]  R.  Sutton,  A.  Barto,  Reinforcement learning: An introduction.  MIT press, 1998. [6]  A. Sykulski, N. Adams, N. Jennings, “On-line Adaptation of Exploration in the One-Armed Bandit with Covariate Problem”, Proc. of 9th Int. Conf. on Machine Learning and Applications, 2010. [7]  H.P.  Schwefel,  Numerical Optimization of Computer Models,  John Wiley and Sons, 1981. Proceedings of EPNC 2012, June 26 - June 29, 2012, Pula, CROATIA 2

Efficient global optimization of expensive black-box functions”,

Exploration versus exploitation using kriging surrogate modelling in electromagnetic design”, accepted to COMPEL, International journal for computation and mathematics in electrical and electronic engineering,

Kriging: a useful tool for electromagnetic device optimization”,

Numerical Optimization of Computer Models,

On the design of optimization strategies based on global response surface approximation models”,

On-line Adaptation of Exploration in the One-Armed Bandit with Covariate Problem”,

Reinforcement learning: An introduction.

Strategies for balancing exploration and exploitation in electromagnetic optimisation

PurposeElectromagnetic design utilising finite element or similar numerical methods is computationally expensive, thus efficient algorithms reducing the number of objective function calls to locate the optimum are sought. The balance between exploration and exploitation may be achieved using a reinforcement learning approach, as demonstrated previously. However, in practical design problems, in addition to finding the global optimum efficiently, information about the robustness of the solution may also be important. In this paper, the aim is to discuss the suitability of different search algorithms and to present their fitness to solve the optimization problem in conjunction with providing enough information on the robustness of the solution.Design/methodology/approachTwo novel strategies enhanced by the surrogate model based weighted expected improvement approach are discussed. The algorithms are tested using a two‐variable test function. The emphasis of these strategies is on accurate approximation of the shape of the objective function to accomplish a robust design.FindingsThe two novel strategies aim to pursue the optimal value of weights for exploration and exploitation throughout the iterative process for better prediction of the shape of the objective function.Originality/valueIt is argued that the proposed strategies based on adaptively tuning weights perform better in predicting the shape of the objective function. Good accuracy of predicting the shape of the objective function is crucial for achieving a robust design.</jats:sec

Song Xiao

Mihai Rotaru

Jan K. Sykulski

Crossref

COMPEL The International Journal for Computation and Mathematics in Electrical and Electronic Engineering

Purpose – Electromagnetic design utilising finite element or similar numerical methods is computationally expensive, thus efficient algorithms reducing the number of objective function calls to locate the optimum are sought. The balance between exploration and exploitation may be achieved using a reinforcement learning approach, as demonstrated previously. However, in practical design problems, in addition to finding the global optimum efficiently, information about the robustness of the solution may also be important. In this paper, the aim is to discuss the suitability of different search algorithms and to present their fitness to solve the optimization problem in conjunction with providing enough information on the robustness of the solution.Design/methodology/approach – Two novel strategies enhanced by the surrogate model based weighted expected improvement approach are discussed. The algorithms are tested using a two-variable test function. The emphasis of these strategies is on accurate approximation of the shape of the objective function to accomplish a robust design.Findings – The two novel strategies aim to pursue the optimal value of weights for exploration and exploitation throughout the iterative process for better prediction of the shape of the objective function.Originality/value – It is argued that the proposed strategies based on adaptively tuning weights perform better in predicting the shape of the objective function. Good accuracy of predicting the shape of the objective function is crucial for achieving a robust design

Sykulski, J. K.

https://eprints.soton.ac.uk/341089/1/EPNC2012p1.pdf

Strategies for balancing exploration and exploitation in electromagnetic optimisation

Abstract

Similar works

Full text

Available Versions

Southampton (e-Prints Soton)

Crossref

Southampton (e-Prints Soton)