SPH Simulations with Reconfigurable Hardware Accelerator

We present a novel approach to accelerate astrophysical hydrodynamical simulations. In astrophysical many-body simulations, GRAPE (GRAvity piPE) system has been widely used by many researchers. However, in the GRAPE systems, its function is completely fixed because specially developed LSI is used as a computing engine. Instead of using such LSI, we are developing a special purpose computing system using Field Programmable Gate Array (FPGA) chips as the computing engine. Together with our developed programming system, we have implemented computing pipelines for the Smoothed Particle Hydrodynamics (SPH) method on our PROGRAPE-3 system. The SPH pipelines running on PROGRAPE-3 system have the peak speed of 85 GFLOPS and in a realistic setup, the SPH calculation using one PROGRAPE-3 board is 5-10 times faster than the calculation on the host computer. Our results clearly shows for the first time that we can accelerate the speed of the SPH simulations of a simple astrophysical phenomena using considerable computing power offered by the hardware.Comment: 27 pages, 13 figures, submitted to PAS

PGPG: An Automatic Generator of Pipeline Design for Programmable GRAPE Systems

We have developed PGPG (Pipeline Generator for Programmable GRAPE), a software which generates the low-level design of the pipeline processor and communication software for FPGA-based computing engines (FBCEs). An FBCE typically consists of one or multiple FPGA (Field-Programmable Gate Array) chips and local memory. Here, the term "Field-Programmable" means that one can rewrite the logic implemented to the chip after the hardware is completed, and therefore a single FBCE can be used for calculation of various functions, for example pipeline processors for gravity, SPH interaction, or image processing. The main problem with FBCEs is that the user need to develop the detailed hardware design for the processor to be implemented to FPGA chips. In addition, she or he has to write the control logic for the processor, communication and data conversion library on the host processor, and application program which uses the developed processor. These require detailed knowledge of hardware design, a hardware description language such as VHDL, the operating system and the application, and amount of human work is huge. A relatively simple design would require 1 person-year or more. The PGPG software generates all necessary design descriptions, except for the application software itself, from a high-level design description of the pipeline processor in the PGPG language. The PGPG language is a simple language, specialized to the description of pipeline processors. Thus, the design of pipeline processor in PGPG language is much easier than the traditional design. For real applications such as the pipeline for gravitational interaction, the pipeline processor generated by PGPG achieved the performance similar to that of hand-written code. In this paper we present a detailed description of PGPG version 1.0.Comment: 24 pages, 6 figures, accepted PASJ 2005 July 2

PROGRAPE-1: A Programmable, Multi-Purpose Computer for Many-Body Simulations

We have developed PROGRAPE-1 (PROgrammable GRAPE-1), a programmable multi-purpose computer for many-body simulations. The main difference between PROGRAPE-1 and "traditional" GRAPE systems is that the former uses FPGA (Field Programmable Gate Array) chips as the processing elements, while the latter rely on the hardwired pipeline processor specialized to gravitational interactions. Since the logic implemented in FPGA chips can be reconfigured, we can use PROGRAPE-1 to calculate not only gravitational interactions but also other forms of interactions such as van der Waals force, hydrodynamical interactions in SPH calculation and so on. PROGRAPE-1 comprises two Altera EPF10K100 FPGA chips, each of which contains nominally 100,000 gates. To evaluate the programmability and performance of PROGRAPE-1, we implemented a pipeline for gravitational interaction similar to that of GRAPE-3. One pipeline fitted into a single FPGA chip, which operated at 16 MHz clock. Thus, for gravitational interaction, PROGRAPE-1 provided the speed of 0.96 Gflops-equivalent. PROGRAPE will prove to be useful for wide-range of particle-based simulations in which the calculation cost of interactions other than gravity is high, such as the evaluation of SPH interactions.Comment: 20 pages with 9 figures; submitted to PAS

A Multi-Core Numerical Framework for Characterizing Flow in Oil Reservoirs

Presented at the SCS Spring Simulation Multi-Conference – SpringSim 2011, April 4-7, 2011 – Boston, USA Awarded Best Paper in the 19th High Performance Computing Symposium and Best Overall Paper at SpringSim 2011.This paper presents a numerical framework that enables scalable, parallel execution of engineering simulations on multi-core, shared memory architectures. Distribution of the simulations is done by selective hash-tabling of the model domain which spatially decomposes it into a number of orthogonal computational tasks. These tasks, the size of which is critical to optimal cache blocking and consequently performance, are then distributed for execution to multiple threads using the previously presented task management algorithm, H-Dispatch. Two numerical methods, smoothed particle hydrodynamics (SPH) and the lattice Boltzmann method (LBM), are discussed in the present work, although the framework is general enough to be used with any explicit time integration scheme. The implementation of both SPH and the LBM within the parallel framework is outlined, and the performance of each is presented in terms of speed-up and efficiency. On the 24-core server used in this research, near linear scalability was achieved for both numerical methods with utilization efficiencies up to 95%. To close, the framework is employed to simulate fluid flow in a porous rock specimen, which is of broad geophysical significance, particularly in enhanced oil recovery

VINE -- A numerical code for simulating astrophysical systems using particles I: Description of the physics and the numerical methods

We present a Fortran 95 code for simulating the evolution of astrophysical systems using particles to represent the underlying fluid flow. The code is designed to be versatile, flexible and extensible, with modular options that can be selected either at compile time or at run time. We include a number of general purpose modules describing a variety of physical processes commonly required in the astrophysical community. The code can be used as an N-body code to evolve a set of particles in two or three dimensions using either a Leapfrog or Runge-Kutta-Fehlberg integrator, with or without individual timesteps for each particle. Particles may interact gravitationally as $N$-body particles, and all or any subset may also interact hydrodynamically, using the Smoothed Particle Hydrodynamic (SPH) method. Massive point particles (`stars') which may accrete nearby SPH or $N$-body particles may also be included. The default free boundary conditions can be replaced by a module to include periodic boundaries. Cosmological expansion may also be included. An interface with special purpose `GRAPE' hardware may also be selected. If available, forces obtained from the GRAPE coprocessors may be transparently substituted for those obtained from the default tree based calculation. The code may be run without modification on single processors or in parallel using OpenMP compiler directives on large scale, shared memory parallel machines. In comparison to the Gadget-2 code of Springel 2005, the gravitational force calculation is $\approx 3.5 - 4.8$ times faster with VINE when run on 8 Itanium~2 processors in an SGI Altix, while producing nearly identical outcomes in our test problems. We present simulations of several test problems, including a merger simulation of two elliptical galaxies with 800000 particles.Comment: Update to revised and newly resubmitted paper version. Code updated to version 1.0.

Acceleration of Astrophysical Simulations with Special Hardware

This work presents the raceSPH and raceGRAV accelerator libraries, designed to interface astrophysical simulations with special-purpose hardware. The raceSPH focuses on the acceleration of Smoothed Particle Hydrodynamics (SPH), a method for approximating force interactions in fluid dynamics. Accelerators used range from vectorizing units on the microprocessors to Field Programmable Gate Arrays (FPGAs) and Graphics Processing Units (GPUs), and speed-ups range from 1.2x to 28x when measured in a synthetic benchmark and from 6x to 19x when used inside astrophysical simulations, for a total wallclock time speed-up of 1.6x to 2.4x, close to the theoretical maximum of 2.5x. The raceGRAV library computes gravitational force with high accuracy and is designed to complement the GRAPE accelerator. In direct summation tests, it provides performance on par with vectorizing units of the processor and comparable to the GRAPE-6 when normalized against number of pipelines. For the development of these libraries, a set of supporting modules were developed, including a PCI driver for modern Linux kernel versions, an MPRACE library for the communication with FPGA boards and a bu er management library for the efficient handling of data transfers