DALiuGE: A Graph Execution Framework for Harnessing the Astronomical Data Deluge

The Data Activated Liu Graph Engine - DALiuGE - is an execution framework for processing large astronomical datasets at a scale required by the Square Kilometre Array Phase 1 (SKA1). It includes an interface for expressing complex data reduction pipelines consisting of both data sets and algorithmic components and an implementation run-time to execute such pipelines on distributed resources. By mapping the logical view of a pipeline to its physical realisation, DALiuGE separates the concerns of multiple stakeholders, allowing them to collectively optimise large-scale data processing solutions in a coherent manner. The execution in DALiuGE is data-activated, where each individual data item autonomously triggers the processing on itself. Such decentralisation also makes the execution framework very scalable and flexible, supporting pipeline sizes ranging from less than ten tasks running on a laptop to tens of millions of concurrent tasks on the second fastest supercomputer in the world. DALiuGE has been used in production for reducing interferometry data sets from the Karl E. Jansky Very Large Array and the Mingantu Ultrawide Spectral Radioheliograph; and is being developed as the execution framework prototype for the Science Data Processor (SDP) consortium of the Square Kilometre Array (SKA) telescope. This paper presents a technical overview of DALiuGE and discusses case studies from the CHILES and MUSER projects that use DALiuGE to execute production pipelines. In a companion paper, we provide in-depth analysis of DALiuGE's scalability to very large numbers of tasks on two supercomputing facilities.Comment: 31 pages, 12 figures, currently under review by Astronomy and Computin

CHILES: HI morphology and galaxy environment at z=0.12 and z=0.17

We present a study of 16 HI-detected galaxies found in 178 hours of observations from Epoch 1 of the COSMOS HI Large Extragalactic Survey (CHILES). We focus on two redshift ranges between 0.108 <= z <= 0.127 and 0.162 <= z <= 0.183 which are among the worst affected by radio frequency interference (RFI). While this represents only 10% of the total frequency coverage and 18% of the total expected time on source compared to what will be the full CHILES survey, we demonstrate that our data reduction pipeline recovers high quality data even in regions severely impacted by RFI. We report on our in-depth testing of an automated spectral line source finder to produce HI total intensity maps which we present side-by-side with significance maps to evaluate the reliability of the morphology recovered by the source finder. We recommend that this become a common place manner of presenting data from upcoming HI surveys of resolved objects. We use the COSMOS 20k group catalogue, and we extract filamentary structure using the topological DisPerSE algorithm to evaluate the \hi\ morphology in the context of both local and large-scale environments and we discuss the shortcomings of both methods. Many of the detections show disturbed HI morphologies suggesting they have undergone a recent interaction which is not evident from deep optical imaging alone. Overall, the sample showcases the broad range of ways in which galaxies interact with their environment. This is a first look at the population of galaxies and their local and large-scale environments observed in HI by CHILES at redshifts beyond the z=0.1 Universe.Comment: 23 pages, 12 figures, 1 interactive 3D figure, accepted to MNRA

GAMA/G10-COSMOS/3D-HST: The 0<z<5 cosmic star-formation history, stellar- and dust-mass densities

We use the energy-balance code MAGPHYS to determine stellar and dust masses, and dust corrected star-formation rates for over 200,000 GAMA galaxies, 170,000 G10-COSMOS galaxies and 200,000 3D-HST galaxies. Our values agree well with previously reported measurements and constitute a representative and homogeneous dataset spanning a broad range in stellar mass (10^8---10^12 Msol), dust mass (10^6---10^9 Msol), and star-formation rates (0.01---100 Msol per yr), and over a broad redshift range (0.0 < z < 5.0). We combine these data to measure the cosmic star-formation history (CSFH), the stellar-mass density (SMD), and the dust-mass density (DMD) over a 12 Gyr timeline. The data mostly agree with previous estimates, where they exist, and provide a quasi-homogeneous dataset using consistent mass and star-formation estimators with consistent underlying assumptions over the full time range. As a consequence our formal errors are significantly reduced when compared to the historic literature. Integrating our cosmic star-formation history we precisely reproduce the stellar-mass density with an ISM replenishment factor of 0.50 +/- 0.07, consistent with our choice of Chabrier IMF plus some modest amount of stripped stellar mass. Exploring the cosmic dust density evolution, we find a gradual increase in dust density with lookback time. We build a simple phenomenological model from the CSFH to account for the dust mass evolution, and infer two key conclusions: (1) For every unit of stellar mass which is formed 0.0065---0.004 units of dust mass is also formed; (2) Over the history of the Universe approximately 90 to 95 per cent of all dust formed has been destroyed and/or ejected