Optimization and Portability of a Fusion OpenACC-based FORTRAN HPC Code from NVIDIA to AMD GPUs

NVIDIA has been the main provider of GPU hardware in HPC systems for over a decade. Most applications that benefit from GPUs have thus been developed and optimized for the NVIDIA software stack. Recent exascale HPC systems are, however, introducing GPUs from other vendors, e.g. with the AMD GPU-based OLCF Frontier system just becoming available. AMD GPUs cannot be directly accessed using the NVIDIA software stack, and require a porting effort by the application developers. This paper provides an overview of our experience porting and optimizing the CGYRO code, a widely-used fusion simulation tool based on FORTRAN with OpenACC-based GPU acceleration. While the porting from the NVIDIA compilers was relatively straightforward using the CRAY compilers on the AMD systems, the performance optimization required more fine-tuning. In the optimization effort, we uncovered code sections that had performed well on NVIDIA GPUs, but were unexpectedly slow on AMD GPUs. After AMD-targeted code optimizations, performance on AMD GPUs has increased to meet our expectations. Modest speed improvements were also seen on NVIDIA GPUs, which was an unexpected benefit of this exercise.Comment: 6 pages, 4 figures, 2 tables, To be published in Proceedings of PEARC2

Steady-State Gyrokinetics Transport Code (SSGKT), A Scientific Application Partnership with the Framework Application for Core-Edge Transport Simulations, Final Report

This project initiated the development of TGYRO ? a steady-state Gyrokinetic transport code (SSGKT) that integrates micro-scale GYRO turbulence simulations into a framework for practical multi-scale simulation of conventional tokamaks as well as future reactors. Using a lightweight master transport code, multiple independent (each massively parallel) gyrokinetic simulations are coordinated. The capability to evolve profiles using the TGLF model was also added to TGYRO and represents a more typical use-case for TGYRO. The goal of the project was to develop a steady-state Gyrokinetic transport code (SSGKT) that integrates micro-scale gyrokinetic turbulence simulations into a framework for practical multi-scale simulation of a burning plasma core ? the International Thermonuclear Experimental Reactor (ITER) in particular. This multi-scale simulation capability will be used to predict the performance (the fusion energy gain, Q) given the H-mode pedestal temperature and density. At present, projections of this type rely on transport models like GLF23, which are based on rather approximate fits to the results of linear and nonlinear simulations. Our goal is to make these performance projections with precise nonlinear gyrokinetic simulations. The method of approach is to use a lightweight master transport code to coordinate multiple independent (each massively parallel) gyrokinetic simulations using the GYRO code. This project targets the practical multi-scale simulation of a reactor core plasma in order to predict the core temperature and density profiles given the H-mode pedestal temperature and density. A master transport code will provide feedback to O(16) independent gyrokinetic simulations (each massively parallel). A successful feedback scheme offers a novel approach to predictive modeling of an important national and international problem. Success in this area of fusion simulations will allow US scientists to direct the research path of ITER over the next two decades. The design of an efficient feedback algorithm is a serious numerical challenge. Although the power source and transport balance coding in the master are standard, it is nontrivial to design a feedback loop that can cope with outputs that are both intermittent and extremely expensive. A prototypical feedback scheme has already been successfully demonstrated for a single global GYRO simulation, although the robustness and efficiency are likely far from optimal. Once the transport feedback scheme is perfected, it could, in principle, be embedded into any of the more elaborate transport codes (ONETWO, TRANSP, and CORSICA), or adopted by other FSP-related multi-scale projects

Elevating zero dimensional global scaling predictions to self-consistent theory-based simulations

We have developed an innovative workflow, STEP-0D, within the OMFIT integrated modelling framework. Through systematic validation against the International Tokamak Physics Activity (ITPA) global H-mode confinement database, we demonstrated that STEP-0D, on average, predicts the energy confinement time with a mean relative error (MRE) of less than 19%. Moreover, this workflow showed promising potential in predicting plasmas for proposed fusion reactors such as ARC, EU-DEMO, and CFETR, indicating moderate H-factors between 0.9 and 1.2. STEP-0D allows theory-based prediction of tokamak scenarios, beginning with zero-dimensional (0D) quantities. The workflow initiates with the PRO-create module, generating physically consistent plasma profiles and equilibrium using the same 0D quantities as the IPB98(y,2) confinement scaling. This sets the starting point for the STEP (Stability, Transport, Equilibrium, and Pedestal) module, which further iterates between theory-based physics models of equilibrium, core transport, and pedestal to yield a self-consistent solution. Given these attributes, STEP-0D not only improves the accuracy of predicting plasma performance but also provides a path towards a novel fusion power plant (FPP) design workflow. When integrated with engineering and costing models within an optimization, this new approach could eliminate the iterative reconciliation between plasma models of varying fidelity. This potential for a more efficient design process underpins STEP-0D's significant contribution to future fusion power plant development.Comment: 12 pages, 13 figures, accepted by Physics of Plasmas 202

Stable Deuterium-Tritium plasmas with improved confinement in the presence of energetic-ion instabilities

Providing stable and clean energy sources is a necessity for the increasing demands of humanity. Energy produced by Deuterium (D) and Tritium (T) fusion reactions, in particular in tokamaks, is a promising path towards that goal. However, there is little experience with plasmas formed by D-T mixtures, since most of the experiments are currently performed in pure D. After more than 20 years, the Joint European Torus (JET) has carried out new D-T experiments with the aim of exploring some of the unique characteristics expected in future fusion reactors, such as the presence of highly energetic ions in low plasma rotation conditions. A new stable, high confinement and impurity-free D-T regime, with reduction of energy losses with respect to D, has been found. Multiscale physics mechanisms critically determine the thermal confinement. These crucial achievements importantly contribute to the establishment of fusion energy generation as an alternative to fossil fuels

Stable Deuterium-Tritium burning plasmas with improved confinement in the presence of energetic-ion instabilities

Providing stable and clean energy sources is a necessity for the increasing demands of humanity. Energy produced by fusion reactions, in particular in tokamaks, is a promising path towards that goal. However, there is little experience with plasmas under conditions close to those expected in future fusion reactors, because it requires the fusion of Deuterium (D) and Tritium (T), while most of the experiments are currently performed in pure D. After more than 20 years, the Joint European Torus (JET) has carried out new D-T experiments with the aim of exploring the unique characteristics of burning D-T plasmas, such as the presence of highly energetic ions. A new stable, high confinement and impurity-free D-T regime, with strong reduction of energy losses with respect to D, has been found. Multiscale physics mechanisms critically determine the thermal confinement and the fusion power yield. These crucial achievements importantly contribute to the establishment of fusion energy generation as an alternative to fossil fuels