47 research outputs found
Data Multiplexing in Radio Interferometric Calibration
New and upcoming radio interferometers will produce unprecedented amounts of
data that demand extremely powerful computers for processing. This is a
limiting factor due to the large computational power and energy costs involved.
Such limitations restrict several key data processing steps in radio
interferometry. One such step is calibration where systematic errors in the
data are determined and corrected. Accurate calibration is an essential
component in reaching many scientific goals in radio astronomy and the use of
consensus optimization that exploits the continuity of systematic errors across
frequency significantly improves calibration accuracy. In order to reach full
consensus, data at all frequencies need to be calibrated simultaneously. In the
SKA regime, this can become intractable if the available compute agents do not
have the resources to process data from all frequency channels simultaneously.
In this paper, we propose a multiplexing scheme that is based on the
alternating direction method of multipliers (ADMM) with cyclic updates. With
this scheme, it is possible to simultaneously calibrate the full dataset using
far fewer compute agents than the number of frequencies at which data are
available. We give simulation results to show the feasibility of the proposed
multiplexing scheme in simultaneously calibrating a full dataset when a limited
number of compute agents are available.Comment: MNRAS Accepted 2017 November 28. Received 2017 November 28; in
original form 2017 July 0
Processing Radio Astronomical Data Using the PROCESS Software Ecosystem
In this paper we discuss our efforts in "unlocking" the Long Term Archive (LTA) of the LOFAR radio telescope using the software ecosystem developed in the PROCESS project. The LTA is a large (>50 PB) archive that expands with about 7 PB per year by the ingestion of new observations. It consists of coarsely calibrated "visibilities", i.e. correlations between signals from LOFAR stations. Converting these observations into sky maps (images), which are needed for astronomy research, can be challenging due to the data sizes of the observations and the complexity and compute requirements of the software involved. Using the PROCESS software environment and testbed, we enable a simple point-and-click-reduction of LOFAR observations into sky maps for users of this archive. This work was performed as part of the PROCESS project which aims to provide generalizable open source solutions for user friendly exascale data processing
matchms - processing and similarity evaluation of mass spectrometry data
Mass spectrometry data is at the heart of numerous applications in the biomedical and lifesciences. With growing use of high-throughput techniques, researchers need to analyze largerand more complex datasets. In particular through joint effort in the research community,fragmentation mass spectrometry datasets are growing in size and number. Platforms such asMassBank (Horai et al., 2010), GNPS (Wang et al., 2016) or MetaboLights (Haug et al., 2020)serve as an open-access hub for sharing of raw, processed, or annotated fragmentation massspectrometry data. Without suitable tools, however, exploitation of such datasets remainsoverly challenging. In particular, large collected datasets contain data acquired using differentinstruments and measurement conditions, and can further contain a significant fraction ofinconsistent, wrongly labeled, or incorrect metadata (annotations)
The LOFAR Transients Pipeline
Current and future astronomical survey facilities provide a remarkably rich
opportunity for transient astronomy, combining unprecedented fields of view
with high sensitivity and the ability to access previously unexplored
wavelength regimes. This is particularly true of LOFAR, a
recently-commissioned, low-frequency radio interferometer, based in the
Netherlands and with stations across Europe. The identification of and response
to transients is one of LOFAR's key science goals. However, the large data
volumes which LOFAR produces, combined with the scientific requirement for
rapid response, make automation essential. To support this, we have developed
the LOFAR Transients Pipeline, or TraP. The TraP ingests multi-frequency image
data from LOFAR or other instruments and searches it for transients and
variables, providing automatic alerts of significant detections and populating
a lightcurve database for further analysis by astronomers. Here, we discuss the
scientific goals of the TraP and how it has been designed to meet them. We
describe its implementation, including both the algorithms adopted to maximize
performance as well as the development methodology used to ensure it is robust
and reliable, particularly in the presence of artefacts typical of radio
astronomy imaging. Finally, we report on a series of tests of the pipeline
carried out using simulated LOFAR observations with a known population of
transients.Comment: 30 pages, 11 figures; Accepted for publication in Astronomy &
Computing; Code at https://github.com/transientskp/tk
Predicting low-frequency radio fluxes from known extrasolar planets
International audienc