New, efficient, and accurate high order derivative and dissipation operators satisfying summation by parts, and applications in three-dimensional multi-block evolutions

We construct new, efficient, and accurate high-order finite differencing operators which satisfy summation by parts. Since these operators are not uniquely defined, we consider several optimization criteria: minimizing the bandwidth, the truncation error on the boundary points, the spectral radius, or a combination of these. We examine in detail a set of operators that are up to tenth order accurate in the interior, and we surprisingly find that a combination of these optimizations can improve the operators' spectral radius and accuracy by orders of magnitude in certain cases. We also construct high-order dissipation operators that are compatible with these new finite difference operators and which are semi-definite with respect to the appropriate summation by parts scalar product. We test the stability and accuracy of these new difference and dissipation operators by evolving a three-dimensional scalar wave equation on a spherical domain consisting of seven blocks, each discretized with a structured grid, and connected through penalty boundary conditions.Comment: 16 pages, 9 figures. The files with the coefficients for the derivative and dissipation operators can be accessed by downloading the source code for the document. The files are located in the "coeffs" subdirector

Non-symmetric trapped surfaces in the Schwarzschild and Vaidya spacetimes

Marginally trapped surfaces (MTSs) are commonly used in numerical relativity to locate black holes. For dynamical black holes, it is not known generally if this procedure is sufficiently reliable. Even for Schwarzschild black holes, Wald and Iyer constructed foliations which come arbitrarily close to the singularity but do not contain any MTSs. In this paper, we review the Wald-Iyer construction, discuss some implications for numerical relativity, and generalize to the well known Vaidya spacetime describing spherically symmetric collapse of null dust. In the Vaidya spacetime, we numerically locate non-spherically symmetric trapped surfaces which extend outside the standard spherically symmetric trapping horizon. This shows that MTSs are common in this spacetime and that the event horizon is the most likely candidate for the boundary of the trapped region.Comment: 4 pages, 3 figures; v2: minor modifications; v3: clarified conclusion

Numerical relativity with characteristic evolution, using six angular patches

The characteristic approach to numerical relativity is a useful tool in evolving gravitational systems. In the past this has been implemented using two patches of stereographic angular coordinates. In other applications, a six-patch angular coordinate system has proved effective. Here we investigate the use of a six-patch system in characteristic numerical relativity, by comparing an existing two-patch implementation (using second-order finite differencing throughout) with a new six-patch implementation (using either second- or fourth-order finite differencing for the angular derivatives). We compare these different codes by monitoring the Einstein constraint equations, numerically evaluated independently from the evolution. We find that, compared to the (second-order) two-patch code at equivalent resolutions, the errors of the second-order six-patch code are smaller by a factor of about 2, and the errors of the fourth-order six-patch code are smaller by a factor of nearly 50.Comment: 12 pages, 5 figures, submitted to CQG (special NFNR issue

Computational Relativistic Astrophysics With Adaptive Mesh Refinement: Testbeds

We have carried out numerical simulations of strongly gravitating systems based on the Einstein equations coupled to the relativistic hydrodynamic equations using adaptive mesh refinement (AMR) techniques. We show AMR simulations of NS binary inspiral and coalescence carried out on a workstation having an accuracy equivalent to that of a $1025^3$ regular unigrid simulation, which is, to the best of our knowledge, larger than all previous simulations of similar NS systems on supercomputers. We believe the capability opens new possibilities in general relativistic simulations.Comment: 7 pages, 16 figure

A Massive Data Parallel Computational Framework for Petascale/Exascale Hybrid Computer Systems

Heterogeneous systems are becoming more common on High Performance Computing (HPC) systems. Even using tools like CUDA and OpenCL it is a non-trivial task to obtain optimal performance on the GPU. Approaches to simplifying this task include Merge (a library based framework for heterogeneous multi-core systems), Zippy (a framework for parallel execution of codes on multiple GPUs), BSGP (a new programming language for general purpose computation on the GPU) and CUDA-lite (an enhancement to CUDA that transforms code based on annotations). In addition, efforts are underway to improve compiler tools for automatic parallelization and optimization of affine loop nests for GPUs and for automatic translation of OpenMP parallelized codes to CUDA. In this paper we present an alternative approach: a new computational framework for the development of massively data parallel scientific codes applications suitable for use on such petascale/exascale hybrid systems built upon the highly scalable Cactus framework. As the first non-trivial demonstration of its usefulness, we successfully developed a new 3D CFD code that achieves improved performance.Comment: Parallel Computing 2011 (ParCo2011), 30 August -- 2 September 2011, Ghent, Belgiu

From Physics Model to Results: An Optimizing Framework for Cross-Architecture Code Generation

Starting from a high-level problem description in terms of partial differential equations using abstract tensor notation, the Chemora framework discretizes, optimizes, and generates complete high performance codes for a wide range of compute architectures. Chemora extends the capabilities of Cactus, facilitating the usage of large-scale CPU/GPU systems in an efficient manner for complex applications, without low-level code tuning. Chemora achieves parallelism through MPI and multi-threading, combining OpenMP and CUDA. Optimizations include high-level code transformations, efficient loop traversal strategies, dynamically selected data and instruction cache usage strategies, and JIT compilation of GPU code tailored to the problem characteristics. The discretization is based on higher-order finite differences on multi-block domains. Chemora's capabilities are demonstrated by simulations of black hole collisions. This problem provides an acid test of the framework, as the Einstein equations contain hundreds of variables and thousands of terms.Comment: 18 pages, 4 figures, accepted for publication in Scientific Programmin

High accuracy binary black hole simulations with an extended wave zone

We present results from a new code for binary black hole evolutions using the moving-puncture approach, implementing finite differences in generalised coordinates, and allowing the spacetime to be covered with multiple communicating non-singular coordinate patches. Here we consider a regular Cartesian near zone, with adapted spherical grids covering the wave zone. The efficiencies resulting from the use of adapted coordinates allow us to maintain sufficient grid resolution to an artificial outer boundary location which is causally disconnected from the measurement. For the well-studied test-case of the inspiral of an equal-mass non-spinning binary (evolved for more than 8 orbits before merger), we determine the phase and amplitude to numerical accuracies better than 0.010% and 0.090% during inspiral, respectively, and 0.003% and 0.153% during merger. The waveforms, including the resolved higher harmonics, are convergent and can be consistently extrapolated to $r\to\infty$ throughout the simulation, including the merger and ringdown. Ringdown frequencies for these modes (to $(\ell,m)=(6,6)$) match perturbative calculations to within 0.01%, providing a strong confirmation that the remnant settles to a Kerr black hole with irreducible mass $M_{\rm irr} = 0.884355\pm20\times10^{-6}$ and spin $S_f/M_f^2 = 0.686923 \pm 10\times10^{-6}