Sofia 2 - An automated, parallel H i source finding pipeline for the WALLABY survey

We present SoFiA 2, the fully automated 3D source finding pipeline for the WALLABY extragalactic HI survey with the Australian SKA Pathfinder (ASKAP). SoFiA 2 is a reimplementation of parts of the original SoFiA pipeline in the C programming language and makes use of OpenMP for multi-threading of the most time-critical algorithms. In addition, we have developed a parallel framework called SoFiA-X that allows the processing of large data cubes to be split across multiple computing nodes. As a result of these efforts, SoFiA 2 is substantially faster and comes with a much reduced memory footprint compared to its predecessor, thus allowing the large WALLABY data volumes of hundreds of gigabytes of imaging data per epoch to be processed in real-time. The source code has been made publicly available to the entire community under an open-source licence. Performance tests using mock galaxies injected into genuine ASKAP data suggest that in the absence of significant imaging artefacts SoFiA 2 is capable of achieving near-100% completeness and reliability above an integrated signal-to-noise ratio of about 5-6. We also demonstrate that SoFiA 2 generally recovers the location, integrated flux and w20 line width of galaxies with high accuracy. Other parameters, including the peak flux density and w50 line width, are more strongly biased due to the influence of the noise on the measurement. In addition, very faint galaxies below an integrated signal-to-noise ratio of about 10 may get broken up into multiple components, thus requiring a strategy to identify fragmented sources and ensure that they do not affect the integrity of any scientific analysis based on the SoFiA 2 output.Comment: 16 pages, 10 figures, 1 table, accepted for publication in MNRA

SKA Science Data Challenge 2: analysis and results

The Square Kilometre Array Observatory (SKAO) will explore the radio sky to new depths in order to conduct transformational science. SKAO data products made available to astronomers will be correspondingly large and complex, requiring the application of advanced analysis techniques to extract key science findings. To this end, SKAO is conducting a series of Science Data Challenges, each designed to familiarise the scientific community with SKAO data and to drive the development of new analysis techniques. We present the results from Science Data Challenge 2 (SDC2), which invited participants to find and characterise 233245 neutral hydrogen (Hi) sources in a simulated data product representing a 2000~h SKA MID spectral line observation from redshifts 0.25 to 0.5. Through the generous support of eight international supercomputing facilities, participants were able to undertake the Challenge using dedicated computational resources. Alongside the main challenge, `reproducibility awards' were made in recognition of those pipelines which demonstrated Open Science best practice. The Challenge saw over 100 participants develop a range of new and existing techniques, with results that highlight the strengths of multidisciplinary and collaborative effort. The winning strategy -- which combined predictions from two independent machine learning techniques to yield a 20 percent improvement in overall performance -- underscores one of the main Challenge outcomes: that of method complementarity. It is likely that the combination of methods in a so-called ensemble approach will be key to exploiting very large astronomical datasets.Comment: Under review by MNRAS; 28 pages, 16 figure

Asymmetric distribution of data products from WALLABY, an SKA precursor neutral hydrogen survey

Póster presentado online al congreso: Astronomical Data Analysis Software and Systems (ADASS) XXXI, celebrado el 24 – 28 October 2021 Cape Town, South Africa and online.- Video de la presentación https://adass2021.ac.za/uploads/X7-009/upload/X7-009_latest.mp4The Widefield ASKAP L-band Legacy All-sky Blind surveY (WALLABY) (1) is a neutral hydrogen survey(HI) that is running on the Australian SKA Pathfinder (ASKAP), a precursor telescope for the Square Kilometre Array(SKA). The goal of WALLABY is to use ASKAP's powerful widefield phased arrayfeed technologyto observe three quarters of the entire sky at the 21 cm neutral hydrogen line with an angular resolution of 30 arcseconds. Post-processing activities at the Australian SKA Regional Centre (AusSRC), Canadian Initiative for Radio Astronomy Data Analysis (CIRADA) and Spanish SKA Regional Centre prototype (SPSRC) will then produce publicly available advanced data products in the form of source catalogues,kinematic models and image cutouts, respectively. This scenario will require the replication and synchronisation of database tables across all sites where a multi-site data repository environment will be essential for efficient distribution management of the access, location and delivery of data.We explore the use of an asymmetric database replication model and strategy, using PostgreSQL as the engine and Bucardo as the asynchronous replication service to distribute data products from WALLABY. This work would serve to evaluate this type of data distribution solution across globally distributed sites and to confirm that the deployed model is sufficient for future scalability and remote collaboration needs

Learning from FITS: Limitations in use in modern astronomical research

The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard, however, is showing its age. Developed in the late 1970s, the FITS authors made a number of implementation choices that, while common at the time, are now seen to limit its utility with modern data. The authors of the FITS standard could not anticipate the challenges which we are facing today in astronomical computing. Difficulties we now face include, but are not limited to, addressing the need to handle an expanded range of specialized data product types (data models), being more conducive to the networked exchange and storage of data, handling very large datasets, and capturing significantly more complex metadata and data relationships. There are members of the community today who find some or all of these limitations unworkable, and have decided to move ahead with storing data in other formats. If this fragmentation continues, we risk abandoning the advantages of broad interoperability, and ready archivability, that the FITS format provides for astronomy. In this paper we detail some selected important problems which exist within the FITS standard today. These problems may provide insight into deeper underlying issues which reside in the format and we provide a discussion of some lessons learned. It is not our intention here to prescribe specific remedies to these issues; rather, it is to call attention of the FITS and greater astronomical computing communities to these problems in the hope that it will spur action to address them

Science with the Murchison Widefield Array: Phase I results and Phase II opportunities

The Murchison Widefield Array (MWA) is an open access telescope dedicated to studying the low-frequency (80-300 MHz) southern sky. Since beginning operations in mid-2013, the MWA has opened a new observational window in the southern hemisphere enabling many science areas. The driving science objectives of the original design were to observe 21 cm radiation from the Epoch of Reionisation (EoR), explore the radio time domain, perform Galactic and extragalactic surveys, and monitor solar, heliospheric, and ionospheric phenomena. All together 60+ programs recorded 20 000 h producing 146 papers to date. In 2016, the telescope underwent a major upgrade resulting in alternating compact and extended configurations. Other upgrades, including digital back-ends and a rapid-response triggering system, have been developed since the original array was commissioned. In this paper, we review the major results from the prior operation of the MWA and then discuss the new science paths enabled by the improved capabilities. We group these science opportunities by the four original science themes but also include ideas for directions outside these categories....FWN – Publicaties zonder aanstelling Universiteit Leide

Science with the Murchison Widefield Array : Phase l results and Phase II opportunities

The Murchison Widefield Array (MWA) is an open access telescope dedicated to studying the low-frequency (80-300 MHz) southern sky. Since beginning operations in mid-2013, the MWA has opened a new observational window in the southern hemisphere enabling many science areas. The driving science objectives of the original design were to observe 21 cm radiation from the Epoch of Reionisation (EoR), explore the radio time domain, perform Galactic and extragalactic surveys, and monitor solar, heliospheric, and ionospheric phenomena. All together programs recorded 20 000 h producing 146 papers to date. In 2016, the telescope underwent a major upgrade resulting in alternating compact and extended configurations. Other upgrades, including digital back-ends and a rapid-response triggering system, have been developed since the original array was commissioned. In this paper, we review the major results from the prior operation of the MWA and then discuss the new science paths enabled by the improved capabilities. We group these science opportunities by the four original science themes but also include ideas for directions outside these categories