72 research outputs found
Towards exascale real-time RFI mitigation
We describe the design and implementation of an extremely scalable real-time
RFI mitigation method, based on the offline AOFlagger. All algorithms scale
linearly in the number of samples. We describe how we implemented the flagger
in the LOFAR real-time pipeline, on both CPUs and GPUs. Additionally, we
introduce a novel simple history-based flagger that helps reduce the impact of
our small window on the data.
By examining an observation of a known pulsar, we demonstrate that our
flagger can achieve much higher quality than a simple thresholder, even when
running in real time, on a distributed system. The flagger works on visibility
data, but also on raw voltages, and beam formed data. The algorithms are
scale-invariant, and work on microsecond to second time scales. We are
currently implementing a prototype for the time domain pipeline of the SKA
central signal processor.Comment: 2016 Radio Frequency Interference (RFI2016) Coexisting with Radio
Frequency Interference, Socorro, New Mexico, USA, October 201
Real-Time Dedispersion for Fast Radio Transient Surveys, using Auto Tuning on Many-Core Accelerators
Dedispersion, the removal of deleterious smearing of impulsive signals by the
interstellar matter, is one of the most intensive processing steps in any radio
survey for pulsars and fast transients. We here present a study of the
parallelization of this algorithm on many-core accelerators, including GPUs
from AMD and NVIDIA, and the Intel Xeon Phi. We find that dedispersion is
inherently memory-bound. Even in a perfect scenario, hardware limitations keep
the arithmetic intensity low, thus limiting performance. We next exploit
auto-tuning to adapt dedispersion to different accelerators, observations, and
even telescopes. We demonstrate that the optimal settings differ between
observational setups, and that auto-tuning significantly improves performance.
This impacts time-domain surveys from Apertif to SKA.Comment: 8 pages, accepted for publication in Astronomy and Computin
Rocket: Efficient and Scalable All-Pairs Computations on Heterogeneous Platforms
All-pairs compute problems apply a user-defined function to each combination
of two items of a given data set. Although these problems present an abundance
of parallelism, data reuse must be exploited to achieve good performance.
Several researchers considered this problem, either resorting to partial
replication with static work distribution or dynamic scheduling with full
replication. In contrast, we present a solution that relies on hierarchical
multi-level software-based caches to maximize data reuse at each level in the
distributed memory hierarchy, combined with a divide-and-conquer approach to
exploit data locality, hierarchical work-stealing to dynamically balance the
workload, and asynchronous processing to maximize resource utilization. We
evaluate our solution using three real-world applications (from digital
forensics, localization microscopy, and bioinformatics) on different platforms
(from a desktop machine to a supercomputer). Results shows excellent efficiency
and scalability when scaling to 96 GPUs, even obtaining super-linear speedups
due to a distributed cache
Deep Learning Assisted Data Inspection for Radio Astronomy
Modern radio telescopes combine thousands of receivers, long-distance
networks, large-scale compute hardware, and intricate software. Due to this
complexity, failures occur relatively frequently. In this work we propose novel
use of unsupervised deep learning to diagnose system health for modern radio
telescopes. The model is a convolutional Variational Autoencoder (VAE) that
enables the projection of the high dimensional time-frequency data to a
low-dimensional prescriptive space. Using this projection, telescope operators
are able to visually inspect failures thereby maintaining system health. We
have trained and evaluated the performance of the VAE quantitatively in
controlled experiments on simulated data from HERA. Moreover, we present a
qualitative assessment of the the model trained and tested on real LOFAR data.
Through the use of a naive SVM classifier on the projected synthesised data, we
show that there is a trade-off between the dimensionality of the projection and
the number of compounded features in a given spectrogram. The VAE and SVM
combination scores between 65% and 90% accuracy depending on the number of
features in a given input. Finally, we show the prototype
system-health-diagnostic web framework that integrates the evaluated model. The
system is currently undergoing testing at the ASTRON observatory
FAIRSECO: An Extensible Framework for Impact Measurement of Research Software
The growing usage of research software in the research community has highlighted the need to recognize and acknowledge the contributions made not only by researchers but also by Research Software Engineers. However, the existing methods for crediting research software and Research Software Engineers have proven to be insufficient. In response, we have developed FAIRSECO, an extensible open source framework with the objective of assessing the impact of research software in research through the evaluation of various factors. The FAIRSECO framework addresses two critical information needs: firstly, it provides potential users of research software with metrics related to software quality and FAIRness. Secondly, the framework provides information for those who wish to measure the success of a project by offering impact data. By exploring the quality and impact of research software, our aim is to ensure that Research Software Engineers receive the recognition they deserve for their valuable contributions
Lightning Talk:"I solemnly pledge" A Manifesto for Personal Responsibility in the Engineering of Academic Software
International audienceSoftware is fundamental to academic research work, both as part of the method and as the result of research. In June 2016 25 people gathered at Schloss Dagstuhl for a week-long Perspectives Workshop and began to develop a manifesto which places emphasis on the scholarly value of academic software and on personal responsibility. Twenty pledges cover the recognition of academic software, the academic software process and the intellectual content of academic software. This is still work in progress. Through this lightning talk, we aim to get feedback and hone these further, as well as to inspire the WSSSPE audience to think about actions they can take themselves rather than actions they want others to take. We aim to publish a more fully developed Dagstuhl Manifesto by December 2016
The brightness and spatial distributions of terrestrial radio sources
Faint undetected sources of radio-frequency interference (RFI) might become visible in long radio observations when they are consistently present over time. Thereby, they might obstruct the detection of the weak astronomical signals of interest. This iss
- …