Towards exascale real-time RFI mitigation

We describe the design and implementation of an extremely scalable real-time RFI mitigation method, based on the offline AOFlagger. All algorithms scale linearly in the number of samples. We describe how we implemented the flagger in the LOFAR real-time pipeline, on both CPUs and GPUs. Additionally, we introduce a novel simple history-based flagger that helps reduce the impact of our small window on the data. By examining an observation of a known pulsar, we demonstrate that our flagger can achieve much higher quality than a simple thresholder, even when running in real time, on a distributed system. The flagger works on visibility data, but also on raw voltages, and beam formed data. The algorithms are scale-invariant, and work on microsecond to second time scales. We are currently implementing a prototype for the time domain pipeline of the SKA central signal processor.Comment: 2016 Radio Frequency Interference (RFI2016) Coexisting with Radio Frequency Interference, Socorro, New Mexico, USA, October 201

Real-Time Dedispersion for Fast Radio Transient Surveys, using Auto Tuning on Many-Core Accelerators

Dedispersion, the removal of deleterious smearing of impulsive signals by the interstellar matter, is one of the most intensive processing steps in any radio survey for pulsars and fast transients. We here present a study of the parallelization of this algorithm on many-core accelerators, including GPUs from AMD and NVIDIA, and the Intel Xeon Phi. We find that dedispersion is inherently memory-bound. Even in a perfect scenario, hardware limitations keep the arithmetic intensity low, thus limiting performance. We next exploit auto-tuning to adapt dedispersion to different accelerators, observations, and even telescopes. We demonstrate that the optimal settings differ between observational setups, and that auto-tuning significantly improves performance. This impacts time-domain surveys from Apertif to SKA.Comment: 8 pages, accepted for publication in Astronomy and Computin

Deep Learning Assisted Data Inspection for Radio Astronomy

Modern radio telescopes combine thousands of receivers, long-distance networks, large-scale compute hardware, and intricate software. Due to this complexity, failures occur relatively frequently. In this work we propose novel use of unsupervised deep learning to diagnose system health for modern radio telescopes. The model is a convolutional Variational Autoencoder (VAE) that enables the projection of the high dimensional time-frequency data to a low-dimensional prescriptive space. Using this projection, telescope operators are able to visually inspect failures thereby maintaining system health. We have trained and evaluated the performance of the VAE quantitatively in controlled experiments on simulated data from HERA. Moreover, we present a qualitative assessment of the the model trained and tested on real LOFAR data. Through the use of a naive SVM classifier on the projected synthesised data, we show that there is a trade-off between the dimensionality of the projection and the number of compounded features in a given spectrogram. The VAE and SVM combination scores between 65% and 90% accuracy depending on the number of features in a given input. Finally, we show the prototype system-health-diagnostic web framework that integrates the evaluated model. The system is currently undergoing testing at the ASTRON observatory

Lightning Talk:"I solemnly pledge" A Manifesto for Personal Responsibility in the Engineering of Academic Software

International audienceSoftware is fundamental to academic research work, both as part of the method and as the result of research. In June 2016 25 people gathered at Schloss Dagstuhl for a week-long Perspectives Workshop and began to develop a manifesto which places emphasis on the scholarly value of academic software and on personal responsibility. Twenty pledges cover the recognition of academic software, the academic software process and the intellectual content of academic software. This is still work in progress. Through this lightning talk, we aim to get feedback and hone these further, as well as to inspire the WSSSPE audience to think about actions they can take themselves rather than actions they want others to take. We aim to publish a more fully developed Dagstuhl Manifesto by December 2016

The brightness and spatial distributions of terrestrial radio sources

Faint undetected sources of radio-frequency interference (RFI) might become visible in long radio observations when they are consistently present over time. Thereby, they might obstruct the detection of the weak astronomical signals of interest. This iss

Using many-core hardware to correlate radio astronomy signals

A recent development in radio astronomy is to replace traditional disheswithmanysmallantennas. Thesignalsarecombinedtoform one large, virtual telescope. The enormous data streams are crosscorrelated to filter out noise. This is especially challenging, since the computational demands grow quadratically withthe number of data streams. Moreover, the correlator is not only computationally intensive, but also very I/O intensive. The LOFAR telescope, for instance, will produce over 100 terabytes per day. The future SKA telescope will even require in the order of exaflops, and petabits/s of I/O. A recent trend is to correlate in software instead of dedicated hardware. This is done to increase flexibility and to reduce development efforts. Examples include e-VLBIand LOFAR. In this paper, we evaluate the correlator algorithm on multi-core CPUsandmany-corearchitectures,suchasNVIDIAandATIGPUs, and the Cell/B.E. The correlator is a streaming, real-time application, and is much more I/O intensive than applications that are typically implementedon many-core hardware today. We compare withtheLOFARproductioncorrelatoronanIBMBlueGene/Psupercomputer. We investigate performance, power efficiency, and programmability. Weidentifyseveralimportant architecturalproblems which cause architectures to perform suboptimally. Our findings areapplicable todata-intensive applications ingeneral. The results show that the processing power and memory bandwidth of current GPUs are highly imbalanced for correlation purposes. WhiletheproductioncorrelatorontheBlueGene/Pachieves asuperb96%ofthetheoreticalpeakperformance, thisisonly14% on ATI GPUs, and 26 % on NVIDIA GPUs. The Cell/B.E. processor, in contrast, achieves an excellent 92%. We found that the Cell/B.E.is also the most energy-efficient solution, it runs the correlator5-7timesmoreenergyefficientlythantheBlueGene/P.The research presented is an important pathfinder for next-generation telescopes. Categories andSubject Descriptors D.1.3 [Programming Techniques]: Concurrent programming