H-AMR: A New GPU-accelerated GRMHD Code for Exascale Computing With 3D Adaptive Mesh Refinement and Local Adaptive Time-stepping

General-relativistic magnetohydrodynamic (GRMHD) simulations have revolutionized our understanding of black-hole accretion. Here, we present a GPU-accelerated GRMHD code H-AMR with multi-faceted optimizations that, collectively, accelerate computation by 2-5 orders of magnitude for a wide range of applications. Firstly, it involves a novel implementation of a spherical-polar grid with 3D adaptive mesh refinement that operates in each of the 3 dimensions independently. This allows us to circumvent the Courant condition near the polar singularity, which otherwise cripples high-res computational performance. Secondly, we demonstrate that local adaptive time-stepping (LAT) on a logarithmic spherical-polar grid accelerates computation by a factor of $\lesssim10$ compared to traditional hierarchical time-stepping approaches. Jointly, these unique features lead to an effective speed of $\sim10^9$ zone-cycles-per-second-per-node on 5,400 NVIDIA V100 GPUs (i.e., 900 nodes of the OLCF Summit supercomputer). We demonstrate its computational performance by presenting the first GRMHD simulation of a tilted thin accretion disk threaded by a toroidal magnetic field around a rapidly spinning black hole. With an effective resolution of $13$,$440\times4$,$608\times8$,$092$ cells, and a total of $\lesssim22$ billion cells and $\sim0.65\times10^8$ timesteps, it is among the largest astrophysical simulations ever performed. We find that frame-dragging by the black hole tears up the disk into two independently precessing sub-disks. The innermost sub-disk rotation axis intermittently aligns with the black hole spin, demonstrating for the first time that such long-sought alignment is possible in the absence of large-scale poloidal magnetic fields.Comment: 10 pages, 5 figures, submitted to MNRAS, for the YouTube playlist, see https://youtu.be/rIOjKUfzcv

Simulation of the Hydrogen Ground State in Stochastic Electrodynamics-2: Inclusion of Relativistic Corrections

In a recent paper the authors studied numerically the hydrogen ground state in stochastic electrodynamics (SED) within the the non-relativistic approximation. In quantum theory the leading non-relativistic corrections to the ground state energy dominate the Lamb shift related to the photon cloud that should cause the quantum-like behaviour of SED. The present work takes these corrections into account in the numerical modelling. It is found that they have little effect; the self-ionisation that occurs without them remains present. It is speculated that the point-charge approximation for the electron is the cause of the failure

H-AMR: A New GPU-accelerated GRMHD Code for Exascale Computing with 3D Adaptive Mesh Refinement and Local Adaptive Time Stepping

General relativistic magnetohydrodynamic (GRMHD) simulations have revolutionized our understanding of black hole accretion. Here, we present a GPU-accelerated GRMHD code H-AMR with multifaceted optimizations that, collectively, accelerate computation by 2-5 orders of magnitude for a wide range of applications. First, it introduces a spherical grid with 3D adaptive mesh refinement that operates in each of the three dimensions independently. This allows us to circumvent the Courant condition near the polar singularity, which otherwise cripples high-resolution computational performance. Second, we demonstrate that local adaptive time stepping on a logarithmic spherical-polar grid accelerates computation by a factor of ≲10 compared to traditional hierarchical time-stepping approaches. Jointly, these unique features lead to an effective speed of ∼109 zone cycles per second per node on 5400 NVIDIA V100 GPUs (i.e., 900 nodes of the OLCF Summit supercomputer). We illustrate H-AMR's computational performance by presenting the first GRMHD simulation of a tilted thin accretion disk threaded by a toroidal magnetic field around a rapidly spinning black hole. With an effective resolution of 13,440 × 4608 × 8092 cells and a total of ≲22 billion cells and ∼0.65 × 108 time steps, it is among the largest astrophysical simulations ever performed. We find that frame dragging by the black hole tears up the disk into two independently precessing subdisks. The innermost subdisk rotation axis intermittently aligns with the black hole spin, demonstrating for the first time that such long-sought alignment is possible in the absence of large-scale poloidal magnetic fields

H-AMR: A New GPU-accelerated GRMHD Code for Exascale Computing with 3D Adaptive Mesh Refinement and Local Adaptive Time Stepping

General relativistic magnetohydrodynamic (GRMHD) simulations have revolutionized our understanding of black hole accretion. Here, we present a GPU-accelerated GRMHD code H-AMR with multifaceted optimizations that, collectively, accelerate computation by 2-5 orders of magnitude for a wide range of applications. First, it introduces a spherical grid with 3D adaptive mesh refinement that operates in each of the three dimensions independently. This allows us to circumvent the Courant condition near the polar singularity, which otherwise cripples high-resolution computational performance. Second, we demonstrate that local adaptive time stepping on a logarithmic spherical-polar grid accelerates computation by a factor of ≲10 compared to traditional hierarchical time-stepping approaches. Jointly, these unique features lead to an effective speed of ∼109 zone cycles per second per node on 5400 NVIDIA V100 GPUs (i.e., 900 nodes of the OLCF Summit supercomputer). We illustrate H-AMR's computational performance by presenting the first GRMHD simulation of a tilted thin accretion disk threaded by a toroidal magnetic field around a rapidly spinning black hole. With an effective resolution of 13,440 × 4608 × 8092 cells and a total of ≲22 billion cells and ∼0.65 × 108 time steps, it is among the largest astrophysical simulations ever performed. We find that frame dragging by the black hole tears up the disk into two independently precessing subdisks. The innermost subdisk rotation axis intermittently aligns with the black hole spin, demonstrating for the first time that such long-sought alignment is possible in the absence of large-scale poloidal magnetic fields