Coherent synchrotron radiation (CSR) is an effect of self-interaction of an electron bunch as it traverses a curved path. It can cause a significant emittance degradation, as well as fragmentation and microbunching. Numerical simulations of the 2D/3D CSR effects have been extremely challenging due to computational bottlenecks associated with calculating retarded potentials via integrating over the history of the bunch. We present a new high-performance 2D, particle-in-cell code which uses massively parallel multicore GPU/GPU platforms to alleviate computational bottlenecks. The code formulates the CSR problem from first principles by using the retarded scalar and vector potentials to compute the self-interaction fields. The speedup due to the parallel implementation on GPU/CPU platforms exceeds three orders of magnitude, thereby bringing a previously intractable problem within reach. The accuracy of the code is verified against analytic 1D solutions (rigid bunch) and semi-analytic 2D solutions for the chirped bunch. Finally, we use the new code in conjunction with a genetic algorithm to optimize the design of a fiducial chicane

Arumugam, K.

Duffin, D.

Godunov, A.

Islam, T.

Ranjan, D.

Sangam, S.

Terzić, B.

Zubair, Mohammad

English

Old Dominion University

Old Dominion UniversityODU Digital CommonsPhysics Faculty Publications Physics2017Simulations of Coherent Synchrotron Radiation onParallel Hybrid GPU/CPU PlatformB. TerzićOld Dominion University, bterzic@odu.eduK. ArumugamOld Dominion UniversityD. DuffinOld Dominion UniversityA. GodunovOld Dominion University, agodunov@odu.eduT. IslamOld Dominion UniversitySee next page for additional authorsFollow this and additional works at: https://digitalcommons.odu.edu/physics_fac_pubsPart of the Computer Sciences Commons, and the Plasma and Beam Physics CommonsThis Conference Paper is brought to you for free and open access by the Physics at ODU Digital Commons. It has been accepted for inclusion inPhysics Faculty Publications by an authorized administrator of ODU Digital Commons. For more information, please contactdigitalcommons@odu.edu.Repository CitationTerzić, B.; Arumugam, K.; Duffin, D.; Godunov, A.; Islam, T.; Ranjan, D.; Sangam, S.; and Zubair, Mohammad, "Simulations ofCoherent Synchrotron Radiation on Parallel Hybrid GPU/CPU Platform" (2017). Physics Faculty Publications. 263.https://digitalcommons.odu.edu/physics_fac_pubs/263Original Publication CitationTerzić, B., Arumugam, K., Duffin, D., Godunov, A., Islam, T., Ranjan, D., ... & Zubair, M. (2017, May). Simulations of CoherentSynchrotron Radiation on Parallel Hybrid GPU/CPU Platform. In Proceedings of the 8th International Particle Accelerator Conference(IPAC'17), Copenhagen, Denmark, May 14-19, 2017 (pp. 3915-3917). https://doi.org/10.18429/JACoW-IPAC2017-THPAB085AuthorsB. Terzić, K. Arumugam, D. Duffin, A. Godunov, T. Islam, D. Ranjan, S. Sangam, and Mohammad ZubairThis conference paper is available at ODU Digital Commons: https://digitalcommons.odu.edu/physics_fac_pubs/263SIMULATION OF COHERENT SYNCHROTRON RADIATION ONPARALLEL HYBRID GPU/CPU PLATFORM ∗B. Terzić†, K. Arumugam, D. Duffin, A. Godunov, T. Islam, D. Ranjan, S. Sangam, M. ZubairOld Dominion University, Norfolk, VA 23529, USAAbstractCoherent synchrotron radiation (CSR) is an effect of self-interaction of an electron bunch as it traverses a curved path.It can cause a significant emittance degradation, as well asfragmentation and microbunching. Numerical simulationsof the 2D/3D CSR effects have been extremely challengingdue to computational bottlenecks associated with calculat-ing retarded potentials via integrating over the history of thebunch. Our new high-performance 2D, particle-in-cell codeuses massively parallel multicore GPU/GPU platforms toalleviate computational bottlenecks. The code formulatesthe CSR problem from first principles by using the retardedscalar and vector potentials to compute the self-interactionfields. The speedup due to the parallel implementation onGPU/CPU platforms exceeds three orders of magnitude,thereby bringing a previously intractable problem withinreach. The accuracy of the code is verified against analytic1D solutions (rigid bunch) and semi-analytic 2D solutionsfor the chirped bunch. Here we use the new code in conjunc-tion with a genetic algorithm and Bayesian optimization tooptimize the design of a fiducial chicane.COHERENT SYNCHROTRONRADIATIONElectromagnetic radiation is the result of the accelerationof charged particles. When a large group of these chargedparticles emit this radiation, the power emitted can be clas-sified into two groups: coherent and incoherent [1]. Thecoherent portion encompasses the lower frequency spec-trum with little variation in wavelength and phase differ-ence. The constructive interference that results exaggeratesthe power allocated to the coherent portion. This coherentportion is called Coherent Synchrotron Radiation (CSR).CSR is proportional to the square of the number of chargedparticles (N2) and inversely to the longitudinal size of thebeam (1/∆s). CSR is a mechanism whereby the beam self-interacts. Namely, the radiation that the beam emitted at aprevious time can interact with the beam at a later time. Thissimultaneous interaction between beam and radiation leadsnon-linear, systemic effects that cannot be ignored. Amongthese effects that are of interest to study are emittance degra-dation and microbunching.CODEAll classical electromagnetic effects are governed byMaxwell’s equations and the Lorentz force equation, but∗ Work supported by the National Science Foundation grant 1535641.† bterzic@odu.eduanalytical solutions to beam behavior consisting of manythousands of charged particles are unfeasible without drasticsimplifying assumptions. Numerical solutions must be im-plemented to quantify beam behavior. However a numericalapproach presents its own limitations: (i) the position andstrength of self-interacting radiation depends on the beamat previous times. Therefore an algorithm attempting tocalculate self interaction needs to allocate the entire beamhistory to memory (high memory usage); (ii) the Lorentzforce equation leads to the subtraction of very large numbersby virtue of the electric and magnetic contributions to theforce (loss of significance); (iii) the large number of particlesinvolved result in an exponential rise in function evaluations,especially for naive implementations (poor scaling). Ourinnovative 2D particle tracking code has been implementedto mitigate all of these issues [2]. The code assumes a pla-nar beam and tracks the phase space distribution (positionand momenta of each particle). The self-interaction of thebeam is quantified in a time-dependent function called theretarded potential. The simulation uses a particle-in-cell(PiC) method.A rough outline of the code is as follows [2]: (1) Samplethe initial distribution function; (2) Bin particles using adeposition scheme; (3) Compute retarded potentials in thegrid frame using both the previous beam history and externalfields; (4) Compute spatial and temporal derivatives of thepotentials; (5) Compute the self-forces on each particle usingthe derivative values; (6) Advance particles in time by a pushfrom calculated self-forces; (7) Loop back to Step 2 until theend of the simulation is reached. The bulk of the executiontime is taken up by Step 3–computation of retarded poten-tials by integrating over the history of the beam. The highprecision and speed of the new CSR code required a designand implementation of a new adaptive multi-dimensionalquadratures on a hybrid GPU/CPU platform [3, 4].ACCELERATOR DESIGNUnderstanding how the geometry and setup of the ac-celerator affect the beam a priori is of great interest. Wecombine two optimization schemes–genetic algorithm (GA)and Bayesian optimization–with our new CSR code in or-der to compute geometries that produce an optimal beamquality.Here we demonstrate a proof of concept for this idea.A GA and a Bayesian optimization were run to determinewhich geometry of a simple three-bend bunch compressor(Fig. 1) results in an optimal transverse emittance and longi-tudinal spread (multi-objective problem). In transitioningform single- to multi-objective problems, a notion of opti-Proceedings of IPAC2017, Copenhagen, Denmark THPAB08505 Beam Dynamics and Electromagnetic FieldsD11 Code Developments and Simulation TechniquesISBN 978-3-95450-182-33915 Copyright©2017CC-BY-3.0andbytherespectiveauthorsmality changes from the single optimal solution (a minimumor amaximum) to a set of non-dominated solutions–so calledPareto-optimal front. A solution A is said to dominate so-lution B if one or more of A’s objectives is better than thecorresponding values for B, and any remaining objectivesare equal to B’s. The dimensionality of the problem is con-strained by the fact that the entrance and exit direction arethe same, thereby reducing the problem to two independentinputs (2D problem). The beam and simulation parametersare given in Tables 1 and 2.Optimization of other properties of the beam (single- ormulti-objective) with a different geometry and restrictionscan be easily obtained by a simple extension of the paradigmpresented here.Figure 1: Layout of the fiducial three-bend chicane. Thesymmetry imposes that the B1 = B3 = −B2 and D1 = D2,thereby making it a 2D problem. The parameters are variedin the range 0 ≤ B1 ≤ 1 and −5 ≤ D1 ≤ −0.1.Table 1: Beam PropertiesQuantity ValueNormalized horizontal emittance 10−9m radInitial energy spread −10.0m−1Uncorrelated energy spread 10−7Table 2: Numerical Simulations PropertiesQuantity ValueGrid resolution 64 × 64Number of simulation particles 200, 000Number of steps along the lattice 106Each individual simulation takes about 240 seconds toexecute on a single GPU (Tesla K40). The execution time onmultiple GPUs scales roughly as the inverse of the numberof GPUs [5, 6].Genetic AlgorithmThe genetic algorithm (GA) is a tool for multi-dimensional, non-linear optimization that takes inspirationfrom the principles of biological evolution [7, and referencestherein]. GAs are well-suited for functions with many in-terrelated variables and specifications. In addition, GAsare highly parallelizable which suits them for computationclusters.The general method for a GA works by evaluating thefunction for a given set of inputs (individuals). These eval-uations are then analyzed for fitness. The individuals withthe most fit outputs are replicated, combined (recombina-tion), and perturbed (mutation) for the next set of individuals(generation). As the algorithm iterates over several genera-tions, the individuals converge to the input that optimizesthe function.GAs also excel in solving multi-objective problems withmore than one output (optimizing multiple quantities simul-taneously). In order to achieve this, the GA determinesif a individual is dominated. A dominated individual isone where another individual’s objectives are all more fit.A multi-objective GA does not solve for a particular inputbut rather a collection of inputs called the Pareto-optimalfront: a non-dominated set of individuals.We use PISA platform developed at ETH Zürich [8].Bayesian OptimizationBayesian optimization is a sequential design strategy forglobal optimization of black-box functions, which does notrequire derivatives [9]. It is an example of machine learningprocess in which information about previous observationsof the function to be optimized is used to determine thenext optimal point. We use GPyOpt python routine [10] toimplement the Bayesian optimization.Bayesian optimization is by design single-objective. Inorder to use it to compute multi-objective Pareto-optimalfronts, the objective functions, say f1 and f2, as we havehere, are combined into a single objective function f as f =c1 f1 + c2 f2, with c1 + c2 = 1, as explained in [11, AppendixA]. In order to produce the front shown in Fig. 2 below,constants are varied in the range c1 ∈ [0, 1] in increments of0.1, yielding a total of 11 directions tangent to the Pareto-optimal front [11]. All non-dominated points compose thePareto-optimal front from the Bayesian optimization.RESULTSOur intent here is to combine our new CSR code withthe two powerful tools of non-linear, multi-dimensionaloptimization–GA Bayesian optimization–and compare theirperformance on equal footing. The results reported in thissection are based on the two optimization tools executinga comparable number of function evaluations. In this case,each function evaluation is a single simulation executed ona GPU, for parameters reported in Tables 1 and 2.Figure 3 shows Pareto-optimal fronts obtained by roughlythe same number of function evaluations with the two op-timizations techniques–GA and Bayesian. The two frontsare close to each other. Their quality is estimated by theirlocation in relation to the two objective functions shown–thebetter front is one that is farther left and down. The twofronts agree extremely well in the low transverse emittance(high longitudinal spread) region; in the intermediate re-THPAB085 Proceedings of IPAC2017, Copenhagen, DenmarkISBN 978-3-95450-182-33916Copyright©2017CC-BY-3.0andbytherespectiveauthors05 Beam Dynamics and Electromagnetic FieldsD11 Code Developments and Simulation Techniquesgion the GA is slightly better, while in the high transverseemittance (low longitudinal spread) region, the Bayesianoptimization performs better.When plotted in search space to show where the solutionsof the two Pareto-optimal fronts lie, it is evident that the twofronts represent vastly different sets of solutions. We willinvestigate why this happens, and under which conditionsthe results from the two optimization techniques converge. 0 5e-06 1e-05 1.5e-05 2e-05 2.5e-05 3e-05 3.5e-05 4e-05 4.5e-05 5e-05 0  5e-12  1e-11 1.5e-11 2e-11 2.5e-11 3e-11 3.5e-11 4e-11Longitudinal spread [m]Transverse emittance [m rad]BayesianGAFigure 2: Pareto-optimal front from a GA with 37 gener-ations and 20 individuals (blue triangles), and Bayesianoptimization with 770 evaluations (red squares). Each simu-lation uses nearly the same number of function evaluations.Figure 3: Pareto-optimal front from Fig. 2 shown in searchspace. Even though the two methods produce similar-looking Pareto fronts in objective function space, in searchspace these solutions look quite different.SUMMARYWe presented the proof of concept in which we demon-strated that combining powerful optimization tools–GA andBayesian optimization–with our new CSR code [2] producesa chicane geometry with optimal properties of the beam atthe exit. In this work, the relevant properties are beam’s lon-gitudinal spread and its transverse emittance, but the samepowerful mechanism can easily be extended to other beamproperties.REFERENCES[1] J. Schwinger, "On radiation by electrons in a betatron,"LBNL, US, LBNL-39088, Jan 1998. Edited and transcribedby M. Furman.[2] B. Terzić, A. Godunov, K. Arumugam, D. Ranjan, andM. Zubair, "High-performance simulations of coherent syn-chrotron radiation on multicore GPU and CPU platforms," inProc. of International Particle Accelerator Conf. (IPAC’15),Richmond, USA, May 2015.[3] K. Arumugam, A. Godunov, D. Ranjan, B. Terzić, andM. Zubair, "An efficient deterministic algorithm for adap-tive multidimensional integration on GPUs," in 42nd Inter-national Conference on Parallel Processing (ICPP), Lyon,France, Oct. 2013.[4] K. Arumugam, A. Godunov, D. Ranjan, B. Terzić, andM. Zubair, "A memory efficient algorithm for adaptive mul-tidimensional integration with multiple GPUs," in Proc. ofHigh Performance Computing (HiPC), Bangalore, Dec. 2013.[5] K. Arumugam, A. Godunov, D. Ranjan, B. Terzić, andM. Zubair, "High fidelity simulation of collective effects inelectron beams using an innovative parallel method," in Sum-mer Computer Simulation Conference, Chicago, July 2015.[6] K. Arumugam, A. Godunov, D. Ranjan, B. Terzić, andM. Zubair, "A memory efficient parallel simulation of elec-tron beam dynamics using GPUs," in 23rd Annual IEEEInternational Conference on High Performance Computing,Hyderabad, India, Dec. 2016.[7] A. Hofler, B. Terzić, M. Kramer, A. Zvezdin, V. Morozov,Y. Roblin, F. Lin, and C. Jarvis, "Innovative applications ofgenetic algorithms to problems in accelerator physics," Phys.Rev. ST Accel. Beams, vol. 16, p. 010101, Jan. 2014.[8] S. Bleuler, M. Laumanns, L. Thiele, and E. Zitzler, "PISA–Aplatform and programming language independent interfacefor search algorithms," ETH Zürich, Institut fur TechnischeInformatik und Kommunikationsnetze ETH Zurich, TechReport 154, 2002.[9] J. Mockus, "Bayesian approach to global optimization: the-ory and applications," Doordrecht, Netherlands: Kluwer Aca-demic, 2013.[10] http://sheffieldml.github.io/GPyOpt/index.html[11] B. Terzić, A. Hofler, C. Reeves, S. Khan, G. Krafft, J. Benesch,A. Freyberger, and D. Ranjan, "Simultaneous optimizationof the cavity heat load and trip rates in linacs using a geneticalgorithm," Phys. Rev. ST Accel. Beams, vol. 17, p. 101003,Oct. 2014.Proceedings of IPAC2017, Copenhagen, Denmark THPAB08505 Beam Dynamics and Electromagnetic FieldsD11 Code Developments and Simulation TechniquesISBN 978-3-95450-182-33917 Copyright©2017CC-BY-3.0andbytherespectiveauthors0.8 0.6 0.4 0.2 \ I "-" .. 'II • r •• • • • • • Bayesian • • . \ . • • • GA .. • ' • • ..:,,.,. 111111 • • • '.ii•~iil~ Wi,i .... r .1"1111 • \ • • • • . . .. ·~ . . . ' . .. .. ••• ,I' • •• .. •• -.. ..., . .,. ~ 0 ~ -~--~-~--~--.. ~-~~ .. ~---"'-=~ ~-~ -5 -4.5 -4 -3 .5 -3 -2.5 D1 -2 -1.5 -1 -0.5 

Simulations of Coherent Synchrotron Radiation on Parallel Hybrid GPU/CPU Platform

https://digitalcommons.odu.edu/cgi/viewcontent.cgi?article=1088&amp;context=physics_fac_pubs

Simulations of Coherent Synchrotron Radiation on Parallel Hybrid GPU/CPU Platform

Abstract

Similar works

Full text

Available Versions

Old Dominion University