H-AMR: A New GPU-accelerated GRMHD Code for Exascale Computing With 3D Adaptive Mesh Refinement and Local Adaptive Time-stepping

General-relativistic magnetohydrodynamic (GRMHD) simulations have revolutionized our understanding of black-hole accretion. Here, we present a GPU-accelerated GRMHD code H-AMR with multi-faceted optimizations that, collectively, accelerate computation by 2-5 orders of magnitude for a wide range of applications. Firstly, it involves a novel implementation of a spherical-polar grid with 3D adaptive mesh refinement that operates in each of the 3 dimensions independently. This allows us to circumvent the Courant condition near the polar singularity, which otherwise cripples high-res computational performance. Secondly, we demonstrate that local adaptive time-stepping (LAT) on a logarithmic spherical-polar grid accelerates computation by a factor of $\lesssim10$ compared to traditional hierarchical time-stepping approaches. Jointly, these unique features lead to an effective speed of $\sim10^9$ zone-cycles-per-second-per-node on 5,400 NVIDIA V100 GPUs (i.e., 900 nodes of the OLCF Summit supercomputer). We demonstrate its computational performance by presenting the first GRMHD simulation of a tilted thin accretion disk threaded by a toroidal magnetic field around a rapidly spinning black hole. With an effective resolution of $13$,$440\times4$,$608\times8$,$092$ cells, and a total of $\lesssim22$ billion cells and $\sim0.65\times10^8$ timesteps, it is among the largest astrophysical simulations ever performed. We find that frame-dragging by the black hole tears up the disk into two independently precessing sub-disks. The innermost sub-disk rotation axis intermittently aligns with the black hole spin, demonstrating for the first time that such long-sought alignment is possible in the absence of large-scale poloidal magnetic fields.Comment: 10 pages, 5 figures, submitted to MNRAS, for the YouTube playlist, see https://youtu.be/rIOjKUfzcv

DIFFERENTIAL EFFECT OF A CHRONIC DOSE OF GAMMA IRRADIATION ON SHRUBS IN THE NORTHERN MOJAVE DESERT.

This report addresses the differential effect of a chronic dose of gamma irradiation on shrubs in the northern Mojave Desert

THE ROLE OF SHRUBS ON REDISTRIBUTION OF MINERAL NUTRIENTS IN SOIL IN THE MOJAVE DESERT

Volume: 4Start Page: 124End Page: 13

Radiation Transport Two-temperature GRMHD Simulations of Warped Accretion Disks

In many black hole (BH) systems, the accretion disk is expected to be misaligned with respect to the BH spin axis. If the scale height of the disk is much smaller than the misalignment angle, the spin of the BH can tear the disk into multiple, independently precessing “sub-disks.” This is most likely to happen during outbursts in black hole X-Ray binaries (BHXRBs) and in active galactic nuclei (AGNs) accreting above a few percent of the Eddington limit, because the disk becomes razor-thin. Disk tearing has the potential to explain variability phenomena including quasi-periodic oscillations in BHXRBs and changing-look phenomena in AGNs. Here, we present the first radiative two-temperature general relativistic magnetohydrodynamic (GRMHD) simulation of a strongly tilted (65°) accretion disk around an M _BH = 10 M _⊙ BH, which tears and precesses. This leads to luminosity swings between a few percent and 50% of the Eddington limit on sub-viscous timescales. Surprisingly, even where the disk is radiation-pressure-dominated, the accretion disk is thermally stable over t ≳ 14,000 r _g / c . This suggests warps play an important role in stabilizing the disk against thermal collapse. The disk forms two nozzle shocks perpendicular to the line of nodes where the scale height of the disk decreases tenfold and the electron temperature reaches T _e ∼ 10 ^8 –10 ^9 K. In addition, optically thin gas crossing the tear between the inner and outer disk gets heated to T _e ∼ 10 ^8 K. This suggests that warped disks may emit a Comptonized spectrum that deviates substantially from idealized models

H-AMR: A New GPU-accelerated GRMHD Code for Exascale Computing with 3D Adaptive Mesh Refinement and Local Adaptive Time Stepping

General relativistic magnetohydrodynamic (GRMHD) simulations have revolutionized our understanding of black hole accretion. Here, we present a GPU-accelerated GRMHD code H-AMR with multifaceted optimizations that, collectively, accelerate computation by 2-5 orders of magnitude for a wide range of applications. First, it introduces a spherical grid with 3D adaptive mesh refinement that operates in each of the three dimensions independently. This allows us to circumvent the Courant condition near the polar singularity, which otherwise cripples high-resolution computational performance. Second, we demonstrate that local adaptive time stepping on a logarithmic spherical-polar grid accelerates computation by a factor of ≲10 compared to traditional hierarchical time-stepping approaches. Jointly, these unique features lead to an effective speed of ∼109 zone cycles per second per node on 5400 NVIDIA V100 GPUs (i.e., 900 nodes of the OLCF Summit supercomputer). We illustrate H-AMR's computational performance by presenting the first GRMHD simulation of a tilted thin accretion disk threaded by a toroidal magnetic field around a rapidly spinning black hole. With an effective resolution of 13,440 × 4608 × 8092 cells and a total of ≲22 billion cells and ∼0.65 × 108 time steps, it is among the largest astrophysical simulations ever performed. We find that frame dragging by the black hole tears up the disk into two independently precessing subdisks. The innermost subdisk rotation axis intermittently aligns with the black hole spin, demonstrating for the first time that such long-sought alignment is possible in the absence of large-scale poloidal magnetic fields

H-AMR: A New GPU-accelerated GRMHD Code for Exascale Computing with 3D Adaptive Mesh Refinement and Local Adaptive Time Stepping

General relativistic magnetohydrodynamic (GRMHD) simulations have revolutionized our understanding of black hole accretion. Here, we present a GPU-accelerated GRMHD code H-AMR with multifaceted optimizations that, collectively, accelerate computation by 2-5 orders of magnitude for a wide range of applications. First, it introduces a spherical grid with 3D adaptive mesh refinement that operates in each of the three dimensions independently. This allows us to circumvent the Courant condition near the polar singularity, which otherwise cripples high-resolution computational performance. Second, we demonstrate that local adaptive time stepping on a logarithmic spherical-polar grid accelerates computation by a factor of ≲10 compared to traditional hierarchical time-stepping approaches. Jointly, these unique features lead to an effective speed of ∼109 zone cycles per second per node on 5400 NVIDIA V100 GPUs (i.e., 900 nodes of the OLCF Summit supercomputer). We illustrate H-AMR's computational performance by presenting the first GRMHD simulation of a tilted thin accretion disk threaded by a toroidal magnetic field around a rapidly spinning black hole. With an effective resolution of 13,440 × 4608 × 8092 cells and a total of ≲22 billion cells and ∼0.65 × 108 time steps, it is among the largest astrophysical simulations ever performed. We find that frame dragging by the black hole tears up the disk into two independently precessing subdisks. The innermost subdisk rotation axis intermittently aligns with the black hole spin, demonstrating for the first time that such long-sought alignment is possible in the absence of large-scale poloidal magnetic fields