Global is an optimization algorithm conceived in the ’80s.

Since then several papers discussed improvements of the algorithm,

but adapting it to a multi-thread execution environment

is only a recent branch of development [1]. Our

previous work focused on parallel implementation on a single

machine but sometimes the use of distributed systems

is inevitable. In this paper we introduce a new version of

Global which is the first step towards a fully distributed algorithm.

While the proposed implementation still works on a

single machine, it is easy to see how gossip based information

sharing can be built into and be utilized by the algorithm.

We show that ParallelGlobal is a feasible way to implement

Global on a distributed system. However, further improvements

must be made to solve real world problems with the

algorithm

Bánhelyi, Balázs

Zombori, Dániel

Repository of the Academy's Library

ParallelGlobal with Low Thread InteractionsDániel ZomboriDepartment of Computational OptimizationUniversity of Szegedzomborid@inf.u-szeged.huDr. Balázs BánhelyiDepartment of Computational OptimizationUniversity of Szegedbanhelyi@inf.u-szeged.huABSTRACTGlobal is an optimization algorithm conceived in the ’80s.Since then several papers discussed improvements of the al-gorithm, but adapting it to a multi-thread execution envi-ronment is only a recent branch of development [1]. Ourprevious work focused on parallel implementation on a sin-gle machine but sometimes the use of distributed systemsis inevitable. In this paper we introduce a new version ofGlobal which is the first step towards a fully distributed algo-rithm. While the proposed implementation still works on asingle machine, it is easy to see how gossip based informationsharing can be built into and be utilized by the algorithm.We show that ParallelGlobal is a feasible way to implementGlobal on a distributed system. However, further improve-ments must be made to solve real world problems with thealgorithm.Categories and Subject Descriptors[Computing methodologies]: Optimization algorithms;[Computing methodologies]: Parallel algorithms1. INTRODUCTIONGlobal is an optimization algorithm built from multiple mod-ules working in an ensemble. While older implementationsviewed the algorithm as a whole, the most recent GlobalJframework handles algorithms as a collection of interlock-ing modules. GlobalJ has several implementations for localsearch algorithms and variants of Global. Main characteris-tics of the single threaded version were established in [4]. Inrecent years Global was further developed [6] and it has sev-eral applications [5, 10] where it aids mostly other researchworks. To speed up optimization processes we developedan algorithm [1] that is capable of utilizing multiple compu-tational threads of a single machine. It cannot be directlyimplemented for distributed systems as the millisecond orderof magnitude latency in communication would significantlyslow down the synchronization of threads. To mitigate thisproblem we propose ParallelGlobal, a parallel implementa-tion suitable for distributed systems with high latency oreven with unreliable communication channels. In this paperwe introduce an experimental version whose main purposeis to test the feasibility of the proposed solution. It providesan algorithm skeleton for a real distributed implementation.2. GLOBALGlobal is a global optimizer designed to solve black box un-constrained optimization problems with low number of func-tion evaluations and probabilistic guarantees [1, 2, 3, 4, 6, 7,8, 11]. It uses local search algorithms to refine multiple sam-ple points hence Global is a multi-start method. Global alsoutilizes the Single Linkage Clustering algorithm to make anestimation about the value of samples from the aspect ofoptimization.2.1 Updated Global AlgorithmWhile the updated Global algorithm has only minor changesand in a lot of cases performs equally to the original, it issuperior in execution order, therefore we consider it as thebasis for improvements.Global has an iterative framework where samples in an it-eration compete with samples of previous iterations. Theoriginal version contains four phases in every iteration con-sisting of sampling, reduction, clustering and local search. Inthe updated algorithm the clustering and local search phasesare merged by an implementation alternating between thetwo.Algorithm 1 describes the updated Global in detail. In lines2-5 the algorithm performs the sampling phase. Selection ofsample points is stochastic, using uniform distribution in thesearch space. The generated samples are placed in containerS which is a list structure. To find the most promisingsamples, S is sorted and a reduced set of samples is acquiredwith the lowest function values. R contains the reduced set,which is removed from S.When samples are ready to be processed, in lines 6-24 thealgorithm alternates between clustering and local searcheswhile there are unprocessed samples left. At 7-15 samplesin R are tried against the clustered samples. To determine ifri ∈ R is part of cluster C we need the distance threshold dc.dc depends on the dimension of the objective function, thenumber of samples currently known in the clustering pro-cess and the α ∈ [0, 1] parameter. The latter controls thedecrease speed of dc while more samples are added, in orderAlgorithm 1 GLOBAL1: while termination-criteria() is not true do2: S ← S∪{ni = uniform(lb, ub) : i ∈ [1, new samples]}3: S ← sort(F (si) < F (si+1)), si ∈ S4: R ← {si : i ∈ [1, reduced set size]}5: S ← S \R6: while R is not ∅ do7: for C in clusters do8: dc ←(1− α 1|clustered|+|R|−1) 1dim(F )9: N ← {ri : dc > ‖ri − cj‖∞ ∧ F (ri) > F (cj)}10: if N is not ∅ then11: C ← C ∪N12: R ← R \N13: repeat iteration14: end if15: end for16: l ← local-search(r1 ∈ R)17: Cl, dmin ← argminC∈clusters∥∥∥∥l − argminci∈C,F (ci)∥∥∥∥∞18: if dmin < dc/10 then19: Cl ← Cl ∪ {l, r1}20: else21: clusters← clusters ∪ {{l, r1}}22: end if23: R ← R \ {r1}24: end while25: end whileto adapt to the expected decrease in distance between tworandom samples. With dc set, sample pairs (ri ∈ R, cj ∈ C)are evaluated to determine if ri is part of C. The two criteriaare having a clustered sample cj with lower function valuethan ri and it being closer with the infinity norm (Man-hattan distance) than dc. Samples in R satisfying both ofthem are moved to the current cluster C. When a sample isclustered, all samples in R can potentially be clustered tootherefore ri ∈ R is rechecked against C. After the for cy-cle finished, samples in R cannot be the part of an existingcluster therefore performing a local search is inevitable.Local searches are performed in lines 16-23, where l is thelocal optimum reached from r1. To determine if l is a newlyfound local optimum a comparison with the cluster centersis needed. The center of a cluster is the sample in the clusterwith the lowest function value. By finding the cluster withthe closest center the algorithm can decide if the optimumis already found. If the distance dmin to the cluster Cl withthe closest center is lower than a tenth of the dc threshold, itis considered the same local optimum. In this case l and r1are added to Cl, otherwise they form a new cluster. Since r1is either in an already existing cluster or in a newly createdone, we can remove it from R. Lines 6-24 are repeated untilR becomes empty. With no unclustered samples left Globalfinished an iteration. The number of executed iterations islimited by the termination criteria.3. PARALLEL GLOBALOur goal is to derive an implementation from the updatedGlobal which is multi-threaded with low interactions be-tween threads. The necessity for low thread interactionscomes from the fact that on huge scale optimization tasks asingle computer is not sufficient and in multi-computer envi-ronments the communication between machines is relativelyslow compared to inter-thread communication. We addressthis problem by removing the synchronization of computa-tional threads and replacing it with a message based infor-mation sharing scheme.We can view ParallelGlobal as a naive parallelization ofGlobal. The main idea lies in the parallel execution of Globaliterations, while sharing information between computationalthreads. Consequently, inter-thread communication is nec-essary, however only a few selected data containers have tobe shared. Also, the shared containers have independentdata points and no deletions, therefore inconsistencies can-not arise from data insertions. These considerations makethe algorithm for distributed systems viable.3.1 ParallelGlobal WorkerAlgorithm 2 describes the ParallelGlobal worker which isthe implementation of a single computational thread. Theworker might run on a machine by itself, or multiple workerscan use the multi-threaded environment of a computer.Algorithm 2 ParallelGlobal1: while termination-criteria() is not true do2: exchange-data()3: s← uniform(lb, ub)4: R ← reduce ({s})5: dc ←(1− α 1|clustered|+1−1) 1dim(F )6: for C in clusters do7: N ← {ri : dc > ‖ri − cj‖∞ ∧ F (ri) > F (cj)}8: C ← C ∪N9: R ← R \N10: end for11: l ← local-search(r1 ∈ R)12: Cl, dmin ← argminC∈clusters∥∥∥∥l − argminci∈CF (ci)∥∥∥∥∞13: if dmin < dc/10 then14: Cl ← Cl ∪ {l, r1}15: else16: clusters← clusters ∪ {{l, r1}}17: end if18: end whileSimilarly to Global, ParallelGlobal also runs in a loop tocomplete iterations until a termination criterion is met. Un-like Global, the new algorithm needs a data exchange step(line 2). At the start of every iteration, received messagescan be processed and new messages can be sent accordingto a suitable policy. The messages contain evaluated datapoints arranged into clusters. These clusters can be handledas if they were evaluated locally by clustering the centerpoint (minimum) of the cluster. If the center point cor-responds to an existing cluster, the two clusters should bemerged while duplicate points are filtered out. Otherwise,the received cluster describes a previously unknown local op-timum and it can be added to the existing clusters withoutmodifications.In lines 3 and 4 happens the sampling and reduction. Inprevious Global versions sampling and reduction was per-formed by taking a randomized sample set, then using asorted sample pool and taking the best samples out. Paral-lelGlobal cannot utilize a common pool efficiently due to thedistributed nature of the system. In this version, for sim-plicity we envisioned taking a single sample every iterationand using stochastic sample reduction, possibly aided withspatial measures on the samples information value. A morecomplex but possible solution would be a distributed samplepool. Samples could be transferred between local pools overreliable data connection. This would ensure that a sample isonly evaluated by a single worker and would create a biggervariety of samples to choose from.In lines 5-10 occurs the clustering. It is very similar to theoriginal clustering algorithm. The only change is that weknow that no more than one sample is in R. This is alsotrue for the local search (lines 11-17) which is identical withthe original local search part.3.2 Current implementationThe current implementation of ParallelGlobal only simulatesthe described functionality with some simplification. First,it runs on a single machine with multiple threads as a sin-gle program. Second, messaging is simulated by synchro-nization on the given containers while they are written, butreading operations happen simultaneously. During cluster-ization, the cluster list is only read to a point determinedbefore the process starts, hence new clusters will be excludedfrom already started searches. This also resembles the effectsof messaging, like delays and losses in information spread.Because no real messaging is present, the exchange-data()function is only a placeholder for now. The reduce() func-tion is also a placeholder and the subject of further develop-ment. Currently, every sample is evaluated by the clusteringand local search steps.4. RESULTSThe algorithm was examined from two aspects; comparisonwith the updated Global in the number of function evalua-tions and scaling of run time with additional threads. Nu-merical results were obtained on the following functions,definitions can be found in [9]. Ackley, Discus, Easom,Griewank, Levy, Rastrigin, Schaffer, Schwefel, Shekel-5 ,Shekel-7 , Shekel-10 , Shubert, Spikes1 and Zakharov. For theevaluations we used two termination criteria, the maximumnumber of function evaluations is 105 which is a soft con-dition therefore overshoot is possible. To check whether anoptimum point is reached we use the following expression|F (x∗)− F (x)| < 10−8 + |F (x∗)| · 10−6where x∗ is a known global optimum point and x is the pointin question. To emulate computationally more expensivefunctions we defined the hardness level. A hardness level ofh means that the function will be evaluated 10h times atthe requested point. Global is a stochastic optimizer, more-over ParallelGlobal is also affected by the operating systemsthread scheduling, consequently run times and the numberof function evaluations can differ largely from one optimiza-tion process to the other. To reduce the noise induced bythis, we obtained data points by averaging the results of 100runs with every configuration. The algorithm parameteriza-tions were identical except for the number of threads.1Spikes function definition:f(x) ={1002 + Πxisin(2pixi), if ‖x− (15.25, 15.75)‖2 > 141000, otherwiseShubert20212223 1  2  4  8  16ParallelGlobal ÷ GlobalNumber of ThreadsRelative NFEV100 hardness101 hardness102 hardness103 hardness2−22−12021 1  2  4  8  16ParallelGlobal ÷ GlobalNumber of ThreadsRelative Speed100 hardness101 hardness102 hardness103 hardnessSpikes202122 1  2  4  8  16ParallelGlobal ÷ GlobalNumber of ThreadsRelative NFEV100 hardness101 hardness102 hardness103 hardness2−52−42−32−22−1202122 1  2  4  8  16ParallelGlobal ÷ GlobalNumber of ThreadsRelative Speed100 hardness101 hardness102 hardness103 hardnessFigure 1: Numeric results on Shubert (left) andSpikes (right) test functions.On the left side of Figure 1 we show results for the Shu-bert test function, namely the number of function evalua-tions (NFEV) and the speed of the optimization process,both relative to Global. On the horizontal axes we see thenumber of threads. The vertical axes show the number offunction evaluations and optimization processes run in unittime respectively, both divided by the result of Global on asingle core. Shubert is a function with many local optimaand a flat global trend. In case of Global, NFEV is mostlyin the [500, 2000] range with an average of 900. On the top-left graph relative NFEV shows that we have an increasewith a factor of two. On a single thread the multiplier of2 shows that the algorithm is by itself inferior to Global.This static multiplier is explained by the lack of a samplepool which reduces the necessary number of local searches.They create the bulk of the NFEV and while Global uses 1.5local searches on average ParallelGlobal needs much more.The dynamic growth is also explained by the local searches,combined with multi-threading. Finding the global opti-mum with local search takes several function evaluations insequence. Since multiple threads start local searches inde-pendently, more evaluations can happen until one of themreaches the global optimum. Moreover when the optimumis found, the program does not terminate immediately, alllocal searches have to finish. This phenomenon increasesthe NFEV due to the intrinsic usage of multi-threading andlocal searches.The bottom-left graph of Figure 1 shows the speedup withadditional threads and different hardness values. While forhardness 0 and 1 the additional threads caused a slowdowndue to synchronization time and increased NFEV, on com-putationally more demanding versions we achieved a signif-icant speedup. The results are promising because for thehardness value of 3 on a single thread a function evaluationtook only 650µs on average. With higher evaluation times,the addition of computational power would have more effect.On the right side of Figure 1, we show the results for theSpikes test function which also has many local optima anda flat global trend. ParallelGlobal suffers from the lack ofa sample pool on the Spikes function too. On the otherhand, no dynamic change in NFEV is experienced. Withouta sample pool, ParallelGlobal had a much harder time find-ing the global optimum, which would often exceed the 105NFEV limit. This resulted in close to constant NFEV andno saturation of threads. Based on the relative speed graph,we gain speed linearly with additional CPU power in everyhardness level. Since the function is very cheap to evaluateand ParallelGlobal has to do much more evaluations, onlyhardness 3 gives an advantage to the multi-threaded imple-mentation.−4−3−2−1 0 1 2ackleydiscus5easomgriew20levyrastrigin−20schaf62schwefel−6sh10 sh5sh7shubertspikeszakh40log 2 ParallelGlobal ÷ GlobalRelative runtimes with 16 threads and hardness 3Figure 2: Relative runtimes on all test functionswith 16 threads and hardness 3.On Figure 2 we show relative runtimes for the configurationof 16 threads and hardness 3 on every test function. Sincethe plot is logarithmic, 0 and values below mean similar andbetter results compared to Global. On the functions whichexperienced slowdown either the lack of a sample pool orthe intrinsic properties of ParallelGlobal prevented gains inspeed. About 50% of the functions with speedup were solvedsuccessfully where the NFEV limit had no effects.5. CONCLUSIONDuring our work we came to multiple important conclu-sions about the ParallelGlobal algorithm. The most neededchange is the implementation of a distributed sample poolwith sample sharing between threads. Having a set of probepoints in the search space would ensure that local searchesonly start from promising regions. This change would prob-ably move the algorithm much closer to the NFEV values ofGlobal.Many of our results show slowdown with ParallelGlobal,but huge improvements as hardness values increase, Shu-bert function is a good example. To keep our run timesmanageable we kept the hardness value relatively low. Bygoing up from the current millisecond order to the secondor 10 second order in function evaluations we would have aclearer image on how much speedup can we achieve. Thiswould still undershoot the evaluation time of many practicalproblems, however it would be sufficient for proper testingon distributed systems.To achieve these changes, first the addition of a distributedframework is needed. Both the sharing of probe samplesand cluster information would rely on it. It is also a key fortesting on computationally expensive problems.6. ACKNOWLEDGMENTSThis work was supported by the Hungarian Governmentunder the grant number EFOP-3.6.1-16-2016-00008. Theproject has been supported by the European Union, co-funded by the European Social Fund, and by the Ja´nosBolyai Research Scholarship of the Hungarian Academy ofSciences.7. REFERENCES[1] B. Ba´nhelyi, T. Csendes, B. Le´vai, L. Pa´l, andD. Zombori. The GLOBAL Optimization Algorithm.Springer, 2018.[2] B. Betro´ and F. Schoen. Optimal and sub-optimalstopping rules for the multistart algorithm in globaloptimization. Mathematical Programming, 57:445–458,1992.[3] C. Boender and A. Rinnooy Kan. On when to stopsampling for the maximum. Global Optimization,1:331–340, 1991.[4] C. Boender, A. Rinnooy Kan, G. Timmer, andL. Stougie. A stochastic method for globaloptimization. Mathematical Programming, 22:125–140,1982.[5] T. Csendes, B. Garay, and B. Ba´nhelyi. A verifiedoptimization technique to locate chaotic regions ofhe´non systems. Journal of Global Optimization,35:145–160, 2006.[6] T. Csendes, L. Pa´l, J. Sendin, and J. Banga. Theglobal optimization method revisited. OptimizationLetters, 2:445–454, 2008.[7] I. Lagaris and I. Tsoulos. Stopping rules forbox-constrained stochastic global optimization. InApplied Mathematics and Computation, 197:622–632,2008.[8] J. Send´ın, J. Banga, and T. Csendes. Extensions of amultistart clustering algorithm for constrained globaloptimization problems. Industrial & EngineeringChemistry Research, 48:3014–3023, 2009.[9] S. Surjanovic and D. Bingham.http://www.sfu.ca/%7essurjano/optimization.html.[10] A. Szenes, B. Ba´nhelyi, L. Z. Szabo´, G. Szabo´,T. Csendes, and M. Csete. Improved emission of sivdiamond color centers embedded into concaveplasmonic core-shell nanoresonators. ScientificReports, 7:an:13845, 2017.[11] A. To¨rn. A search clustering approach to globaloptimization, pages 49–62. Elsevier, North-Holland,1978.

ParallelGlobal with Low Thread Interactions

http://real.mtak.hu/99757/1/MATCOS_2019_paper_19.pdf

ParallelGlobal with Low Thread Interactions

Abstract

Similar works

Full text

Available Versions

Repository of the Academy's Library