A GPU-based Correlator X-engine Implemented on the CHIME Pathfinder

We present the design and implementation of a custom GPU-based compute cluster that provides the correlation X-engine of the CHIME Pathfinder radio telescope. It is among the largest such systems in operation, correlating 32,896 baselines (256 inputs) over 400MHz of radio bandwidth. Making heavy use of consumer-grade parts and a custom software stack, the system was developed at a small fraction of the cost of comparable installations. Unlike existing GPU backends, this system is built around OpenCL kernels running on consumer-level AMD GPUs, taking advantage of low-cost hardware and leveraging packed integer operations to double algorithmic efficiency. The system achieves the required 105TOPS in a 10kW power envelope, making it among the most power-efficient X-engines in use today.Comment: 6 pages, 5 figures. Accepted by IEEE ASAP 201

Canadian Hydrogen Intensity Mapping Experiment (CHIME) Pathfinder

A pathfinder version of CHIME (the Canadian Hydrogen Intensity Mapping Experiment) is currently being commissioned at the Dominion Radio Astrophysical Observatory (DRAO) in Penticton, BC. The instrument is a hybrid cylindrical interferometer designed to measure the large scale neutral hydrogen power spectrum across the redshift range 0.8 to 2.5. The power spectrum will be used to measure the baryon acoustic oscillation (BAO) scale across this poorly probed redshift range where dark energy becomes a significant contributor to the evolution of the Universe. The instrument revives the cylinder design in radio astronomy with a wide field survey as a primary goal. Modern low-noise amplifiers and digital processing remove the necessity for the analog beamforming that characterized previous designs. The Pathfinder consists of two cylinders 37\,m long by 20\,m wide oriented north-south for a total collecting area of 1,500 square meters. The cylinders are stationary with no moving parts, and form a transit instrument with an instantaneous field of view of $\sim$100\,degrees by 1-2\,degrees. Each CHIME Pathfinder cylinder has a feedline with 64 dual polarization feeds placed every $\sim$30\,cm which Nyquist sample the north-south sky over much of the frequency band. The signals from each dual-polarization feed are independently amplified, filtered to 400-800\,MHz, and directly sampled at 800\,MSps using 8 bits. The correlator is an FX design, where the Fourier transform channelization is performed in FPGAs, which are interfaced to a set of GPUs that compute the correlation matrix. The CHIME Pathfinder is a 1/10th scale prototype version of CHIME and is designed to detect the BAO feature and constrain the distance-redshift relation.Comment: 20 pages, 12 figures. submitted to Proc. SPIE, Astronomical Telescopes + Instrumentation (2014

HIRAX:A Probe of Dark Energy and Radio Transients

The Hydrogen Intensity and Real-time Analysis eXperiment (HIRAX) is a new 400-800MHz radio interferometer under development for deployment in South Africa. HIRAX will comprise 1024 six meter parabolic dishes on a compact grid and will map most of the southern sky over the course of four years. HIRAX has two primary science goals: to constrain Dark Energy and measure structure at high redshift, and to study radio transients and pulsars. HIRAX will observe unresolved sources of neutral hydrogen via their redshifted 21-cm emission line (`hydrogen intensity mapping'). The resulting maps of large-scale structure at redshifts 0.8-2.5 will be used to measure Baryon Acoustic Oscillations (BAO). HIRAX will improve upon current BAO measurements from galaxy surveys by observing a larger cosmological volume (larger in both survey area and redshift range) and by measuring BAO at higher redshift when the expansion of the universe transitioned to Dark Energy domination. HIRAX will complement CHIME, a hydrogen intensity mapping experiment in the Northern Hemisphere, by completing the sky coverage in the same redshift range. HIRAX's location in the Southern Hemisphere also allows a variety of cross-correlation measurements with large-scale structure surveys at many wavelengths. Daily maps of a few thousand square degrees of the Southern Hemisphere, encompassing much of the Milky Way galaxy, will also open new opportunities for discovering and monitoring radio transients. The HIRAX correlator will have the ability to rapidly and eXperimentciently detect transient events. This new data will shed light on the poorly understood nature of fast radio bursts (FRBs), enable pulsar monitoring to enhance long-wavelength gravitational wave searches, and provide a rich data set for new radio transient phenomena searches. This paper discusses the HIRAX instrument, science goals, and current status.Comment: 11 pages, 5 figure

Real-time stream processing in radio astronomy

A major challenge in modern radio astronomy is dealing with the massive data volumes generated by wide-bandwidth receivers. Such massive data rates are often too great for a single device to cope, and so processing must be split across multiple devices working in parallel. These devices must work in unison to process incoming data in real time, reduce the data volume to a manageable size, and output a science-ready data product. The aim of this chapter is to give a broad overview of how digital systems for radio telescopes are commonly implemented, with a focus on real-time stream processing over multiple compute devices.Comment: Chapter to appear in "Big Data in Radio Astronomy: Scientific Data Processing for Advanced Radio Telescopes

A GPU based X-Engine for the MeerKAT Radio Telescope

The correlator is a key component of the digital backend of a modern radio telescope array. The 64 antenna MeerKAT telescope has an FX architecture correlator consisting of 64 F-Engines and 256 X-Engines. These F- and X-Engines are all hosted on 128 custom designed FPGA processing boards. This custom board is known as a SKARAB. One SKARAB X-Engine board hosts four logical X-Engines. This SKARAB ingests data at 27.2 Gbps over a 40 GbE connection. It correlates this data in real time. GPU technology has improved significantly since SKARAB was designed. GPUs are now becoming viable alternatives to FPGAs in high performance streaming applications. The objective of this dissertation is to investigate how to build a GPU drop-in replacement X-Engine for MeerKAT and to compare this implementation to a SKARAB X-Engine. This includes the construction and analysis of a prototype GPU X-Engine. The 40 GbE ingest, GPU correlation algorithm and the software pipeline framework that links these two together were identified as the three main sub-systems to focus on in this dissertation. A number of different tools implementing these sub-systems were examined with the most suitable ones being chosen for the prototype. A prototype dual socket system was built that could process the equivalent of two SKARABs worth of X-Engine data. This prototype has two 40 GbE Mellanox NICS running the SPEAD2 library and a single Nvidia GeForce 1080Ti GPU running the xGPU library. A custom pipeline framework built on top of the Intel Threaded Building Blocks (TBB) library was designed to facilitate the ow of data between these sub-systems. The prototype system was compared to two SKARABs. For an equivalent amount of processing, the GPU X-Engine cost R143 000 while the two SKARABs cost R490 000. The power consumption of the GPU X-Engine was more than twice that of the SKARABs (400W compared 180W), while only requiring half as much rack space. GPUs as X-Engines were found to be more suitable than FPGAs when cost and density are the main priorities. When power consumption is the priority, then FPGAs should be used. When running eight logical X-Engines, 85% of the prototype's CPU cores were used while only 75% of the GPU's compute capacity was utilised. The main bottleneck on the GPU X-Engine was on the CPU side of the server. This report suggests that the next iteration of the system should offload some CPU side processing to the GPU and double the number of 40 GbE ports. This could potentially double the system throughput. When considering methods to improve this system, an FPGA/GPU hybrid X-Engine concept was developed that would combine the power saving advantage of FPGAs and the low cost to compute ratio of GPUs

Pathfinding Fast Radio Bursts Localizations using Very Long Baseline Interferometry

Fast radio bursts (FRBs) are millisecond-duration, bright radio transients of extragalactic origin. The Canadian Hydrogen Intensity Mapping Experiment (CHIME) telescope’s CHIME/FRB instrument and other radio telescopes across the globe have detected hundreds of FRBs. Their origins are a mystery. Precise localization within the host is critical to distinguish between progenitor models. This can be achieved through Very Long Baseline Interferometry (VLBI). Until now, VLBI localizations have only been carried out in targeted follow-up observations of some repeating sources which comprise a small fraction of the FRBs. For this work, an interferometric array of 6m dishes was constructed at the Green Bank Observatory as a pathfinder to develop the necessary systems, technology, and techniques to enable VLBI on FRBs. This array called TONE has 8 instrumented dishes and works as a VLBI outrigger for CHIME on a \SI{\sim3300}{\kilo\meter} baseline. This involved construction, commissioning, and integration of the custom analog chains and digital system. TONE is pointed to shadow a portion of the CHIME primary beam at a fixed declination of \SI{22}{\deg}. Upon detection of a single dispersed pulse such as an FRB or a giant pulse from the Crab pulsar, CHIME alerts TONE, triggering a recording of buffered data to disk. In addition to TONE, a single 10-m dish at Algonquin Radio Observatory (ARO10) is set up with a similar infrastructure. Together they form the pathfinders for conducting VLBI for FRBs. We used these VLBI pathfinders to localize FRB 20210603A at the time of detection. The baseband data from CHIME and TONE are used to synthesize single beams at each telescope. The single-beam data from TONE and data from ARO10 are each cross-correlated with the single beam data from CHIME. We use the Crab pulsar for astrometric calibration and additionally correct for clock errors. The calibrated and corrected cross-correlated data is sampled with a likelihood function of the sky location and ionospheric effects using a Markov Chain Monte Carlo method to estimate the Right Ascension and Declination of the FRB. We localize the burst to SDSS J004105.82+211331.9, an edge-on quiescent lenticular galaxy at redshift z $\approx 0.177$. The localization, dispersion measure, rotation measure (RM), and temporal broadening are consistent with an observed line-of-sight through the host galactic disk, suggesting a progenitor from a population coincident with the host galactic plane. The development of the TONE telescope has enabled the localization of the FRB within the host. This is a key stepping stone towards constraining the origins and host environments of FRBs