DALiuGE: A Graph Execution Framework for Harnessing the Astronomical Data Deluge

The Data Activated Liu Graph Engine - DALiuGE - is an execution framework for processing large astronomical datasets at a scale required by the Square Kilometre Array Phase 1 (SKA1). It includes an interface for expressing complex data reduction pipelines consisting of both data sets and algorithmic components and an implementation run-time to execute such pipelines on distributed resources. By mapping the logical view of a pipeline to its physical realisation, DALiuGE separates the concerns of multiple stakeholders, allowing them to collectively optimise large-scale data processing solutions in a coherent manner. The execution in DALiuGE is data-activated, where each individual data item autonomously triggers the processing on itself. Such decentralisation also makes the execution framework very scalable and flexible, supporting pipeline sizes ranging from less than ten tasks running on a laptop to tens of millions of concurrent tasks on the second fastest supercomputer in the world. DALiuGE has been used in production for reducing interferometry data sets from the Karl E. Jansky Very Large Array and the Mingantu Ultrawide Spectral Radioheliograph; and is being developed as the execution framework prototype for the Science Data Processor (SDP) consortium of the Square Kilometre Array (SKA) telescope. This paper presents a technical overview of DALiuGE and discusses case studies from the CHILES and MUSER projects that use DALiuGE to execute production pipelines. In a companion paper, we provide in-depth analysis of DALiuGE's scalability to very large numbers of tasks on two supercomputing facilities.Comment: 31 pages, 12 figures, currently under review by Astronomy and Computin

Improved Techniques for the Surveillance of the Near Earth Space Environment with the Murchison Widefield Array

In this paper we demonstrate improved techniques to extend coherent processing intervals for passive radar processing, with the Murchison Widefield Array. Specifically, we apply a two stage linear range and Doppler migration compensation by utilising Keystone Formatting and a recent dechirping method. These methods are used to further demonstrate the potential for the surveillance of space with the Murchison Widefield Array using passive radar, by detecting objects orders of magnitude smaller than previous work. This paper also demonstrates how the linear Doppler migration methods can be extended to higher order compensation to further increase potential processing intervals.Comment: Presented at the 2019 IEEE Radar Conference in Boston earlier this yea

Landscape and Roadmap of Future Internet and Smart Cities

FP7 Fireball coordination Action, http://www.fireball4smartcities.eu/This final D2.1 report forms a synthesis and further extension of the previousreports D1.2 [M6] and D1.2 [M12]. The key topics addressed in this reportreflect the key priorities of the WP2 and are: * Understanding the Smart City, providing state of the art and trends .FIREBALL understands Smart Cities as innovation ecosystems for the FutureInternet. The three areas of Smart Cities, Future Internet and Living Labsare explored including their interlinkages and how they can be exploited.This results into a mapping of the new landscape of Smart Cities and theFuture Internet. * Smart City case studies . Seven cases have been elaborated as a means toexplore and examine current developments, objectives, strategies in "smartcities" and establish collaboration between Smart Cities and the Eurocitiescommunity on one side and Future Internet and Living Labs on the other. * Collaboration models for Smart Cities innovation. In particular we focuson collaboration models that are fundamental to developing andimplementing common innovation activities of the three communitiesconstituting the FIREBALL domain: Smart Cities, Future Internet and Livinglabs. * Thematic Roadmap of Future Internet and Living Labs for SmartCities . This activity forms input for WP3 activities as well as to the Horizon2020 development process supported by the FISA group of Future InternetSupport Actions. The Roadmap activities also support the development of astrategy to implement collaboration models mentioned.The work regarding collaboration models relates strongly to the companionD1.2 (M12) report on Common Assets and the D1.3 (M12) report on Accessmechanisms. The D1.2 report identifies and describes smart cities, living labsand future Internet common assets, which is fundamental to the collaborationmodels mentioned. The D1.3 develops approaches to create access to theseassets and proposes sharing mechanisms.The topics addressed should be considered in close relation to the communitybuilding and collaborative activities that we have undertaken jointly with FIAand Eurocities communities since starting this project in 2010. Our intentionhas always been not only to produce reports but to play a proactive role inchanging the research and innovation landscape as regards Future Internet,Living Labs and Smart Cities

Smart Cities as Innovation Ecosystems sustained by the Future Internet

FIREBALL White paperThe White Paper focuses on how European cities are currently developing strategies towards becoming "smarter cities" and the lessons we can draw for the future. Such strategies are based on an assessment of the future needs of cities and innovative usages of ICTs embodied in the broadband Internet and Internet-based applications now and foreseen for the future. These strategies are also based on a new understanding of innovation, grounded in the concept of open innovation ecosystems, global innovation chains, and on citizens' empowerment for shaping innovation and urban development. This White Paper is one of the main outcomes of the FIREBALL project (www.fireball4smartcities.eu), a Coordination Action within the 7th Framework Programme for ICT, running in the period 2010-2012. The aim of FIREBALL is to bring together communities and stakeholders who are active in three areas: (1) research and experimentation on the Future Internet (FIRE); (2) open and user-driven innovation (Living Labs); and (3) urban development. The goal is to develop a common vision and a common view on how the different approaches, methodologies, policies and technologies in these areas can be aligned to boost innovation and socio-economic development of cities. The White Paper has explored the landscape of "smart cities" as environments of open and user driven innovation sustained by Future Internet technologies and services. Smart cities are also seen as environments enabled by advanced ICT infrastructure for testing and validating current Future Internet research and experimentation. Overall, the smart city is built upon a triangle of "City" - "Open Innovation Ecosystems" - "Future Internet" components. The White Paper explores also how cities and urban areas represent a critical mass when it comes to shaping the demand for advanced Internet-based services in large-scale testing and validation. Shaping this demand informs ongoing research, experimentation and deployment activities related to Future Internet testbeds, and helps establishing a dialogue between the different communities involved in the development of the future Internet and user-driven environments, to form partnerships and assess social and economic benefits and discovery of migration paths at early stages. Based on a holistic instead of technology merely driven perspective on smart cities, we consider necessary to revisit the concept of the Smart City itself. The concept of the smart city that emerges from FIREBALL can be summarized as follows: "The smart city concept is multi-dimensional. It is a future scenario (what to achieve), even more it is an urban development strategy (how to achieve it). It focuses on how (Internet-related) technologies enhance the lives of citizens. This should not be interpreted as drawing the smart city technology scenario. Rather, the smart city is how citizens are shaping the city in using this technology, and how citizens are enabled to do so. The smart city is about how people are empowered, through using technology, for contributing to urban change and realizing their ambitions. The smart city provides the conditions and resources for change. In this sense, the smart city is an urban laboratory, an urban innovation ecosystem, a living lab, an agent of change. Much less do we see a smart city in terms of a Ranking. This ranking is a moment in time, a superficial result of underlying changes, not the mechanism of transformation. The smart city is the engine of transformation, a generator of solutions for wicked problems, it is how the city is behaving smart.

Multi-messenger observations of a binary neutron star merger

On 2017 August 17 a binary neutron star coalescence candidate (later designated GW170817) with merger time 12:41:04 UTC was observed through gravitational waves by the Advanced LIGO and Advanced Virgo detectors. The Fermi Gamma-ray Burst Monitor independently detected a gamma-ray burst (GRB 170817A) with a time delay of ~1.7 s with respect to the merger time. From the gravitational-wave signal, the source was initially localized to a sky region of 31 deg2 at a luminosity distance of 40+8-8 Mpc and with component masses consistent with neutron stars. The component masses were later measured to be in the range 0.86 to 2.26 Mo. An extensive observing campaign was launched across the electromagnetic spectrum leading to the discovery of a bright optical transient (SSS17a, now with the IAU identification of AT 2017gfo) in NGC 4993 (at ~40 Mpc) less than 11 hours after the merger by the One- Meter, Two Hemisphere (1M2H) team using the 1 m Swope Telescope. The optical transient was independently detected by multiple teams within an hour. Subsequent observations targeted the object and its environment. Early ultraviolet observations revealed a blue transient that faded within 48 hours. Optical and infrared observations showed a redward evolution over ~10 days. Following early non-detections, X-ray and radio emission were discovered at the transient’s position ~9 and ~16 days, respectively, after the merger. Both the X-ray and radio emission likely arise from a physical process that is distinct from the one that generates the UV/optical/near-infrared emission. No ultra-high-energy gamma-rays and no neutrino candidates consistent with the source were found in follow-up searches. These observations support the hypothesis that GW170817 was produced by the merger of two neutron stars in NGC4993 followed by a short gamma-ray burst (GRB 170817A) and a kilonova/macronova powered by the radioactive decay of r-process nuclei synthesized in the ejecta

Common Assets identification and Characterisation

FP7 Fireball coordination Action, http://www.fireball4smartcities.eu/Smart Cities innovation ecosystems are based on an infrastructure forinnovation. This infrastructure is built upon what within FIREBALL we are calling "common assets": resources of various types that can be selected, combined,shared and used by those who want to engage in living labs innovation projects.In this report, common assets types such as technical infrastructures, userdriven open innovation methods and tools, test bed facilities and usercommunities are identified and characterized based on a number of cases. Thecharacterization results into an overview and analysis how configurations of suchcommon assets can be tailored to the needs and requirements of Smart Cities.The work reported in this deliverable forms the basis for describing mechanismsfor access and governance of the common assets (D1.3). Also it is of relevanceto, and has been developed in close coordination with, the Landscape andRoadmap as developed in D2.1, which contains a number of "Smart City" casestudies where common assets are being studied as well.The work presented in this deliverable has been subject of several FIREBALLworkshops where the concept of common assets has been discussed andelaborated: * Future Interne Assembly (FIA) conference, Ghent, December 2010, workshopon "Smart Cities and Future Internet Experimentation". Workshop jointlyorganized with FIRESTATION. * Future Internet Assembly (FIA) conference, Budapest, May 2011, workshopon "Smart Cities and FIRE: Experimentation and Living Labs for the FutureInternet". Workshop jpointly organized with FIRESTATION. * ICE 2011 conference, June 2011, Aachen: workshop "Common Assets forSmart Cities Living Labs Innovation: IT Infrastructures, methods and usercommunities".The work has been subject of elaborate discussions with FIA members and alsohas resulted in a joint proposal for a Support Action in FP7-ICT (currently underevaluation)

Smart Cities as Innovation Ecosystems sustained by the Future Internet

FIREBALL White paperThe White Paper focuses on how European cities are currently developing strategies towards becoming "smarter cities" and the lessons we can draw for the future. Such strategies are based on an assessment of the future needs of cities and innovative usages of ICTs embodied in the broadband Internet and Internet-based applications now and foreseen for the future. These strategies are also based on a new understanding of innovation, grounded in the concept of open innovation ecosystems, global innovation chains, and on citizens' empowerment for shaping innovation and urban development. This White Paper is one of the main outcomes of the FIREBALL project (www.fireball4smartcities.eu), a Coordination Action within the 7th Framework Programme for ICT, running in the period 2010-2012. The aim of FIREBALL is to bring together communities and stakeholders who are active in three areas: (1) research and experimentation on the Future Internet (FIRE); (2) open and user-driven innovation (Living Labs); and (3) urban development. The goal is to develop a common vision and a common view on how the different approaches, methodologies, policies and technologies in these areas can be aligned to boost innovation and socio-economic development of cities. The White Paper has explored the landscape of "smart cities" as environments of open and user driven innovation sustained by Future Internet technologies and services. Smart cities are also seen as environments enabled by advanced ICT infrastructure for testing and validating current Future Internet research and experimentation. Overall, the smart city is built upon a triangle of "City" - "Open Innovation Ecosystems" - "Future Internet" components. The White Paper explores also how cities and urban areas represent a critical mass when it comes to shaping the demand for advanced Internet-based services in large-scale testing and validation. Shaping this demand informs ongoing research, experimentation and deployment activities related to Future Internet testbeds, and helps establishing a dialogue between the different communities involved in the development of the future Internet and user-driven environments, to form partnerships and assess social and economic benefits and discovery of migration paths at early stages. Based on a holistic instead of technology merely driven perspective on smart cities, we consider necessary to revisit the concept of the Smart City itself. The concept of the smart city that emerges from FIREBALL can be summarized as follows: "The smart city concept is multi-dimensional. It is a future scenario (what to achieve), even more it is an urban development strategy (how to achieve it). It focuses on how (Internet-related) technologies enhance the lives of citizens. This should not be interpreted as drawing the smart city technology scenario. Rather, the smart city is how citizens are shaping the city in using this technology, and how citizens are enabled to do so. The smart city is about how people are empowered, through using technology, for contributing to urban change and realizing their ambitions. The smart city provides the conditions and resources for change. In this sense, the smart city is an urban laboratory, an urban innovation ecosystem, a living lab, an agent of change. Much less do we see a smart city in terms of a Ranking. This ranking is a moment in time, a superficial result of underlying changes, not the mechanism of transformation. The smart city is the engine of transformation, a generator of solutions for wicked problems, it is how the city is behaving smart.

The Phase II Murchison Widefield Array : design overview

We describe the motivation and design details of the 'Phase II' upgrade of the Murchison Widefield Array radio telescope. The expansion doubles to 256 the number of antenna tiles deployed in the array. The new antenna tiles enhance the capabilities of the Murchison Widefield Array in several key science areas. Seventy-two of the new tiles are deployed in a regular configuration near the existing array core. These new tiles enhance the surface brightness sensitivity of the array and will improve the ability of the Murchison Widefield Array to estimate the slope of the Epoch of Reionisation power spectrum by a factor of -1/43.5. The remaining 56 tiles are deployed on long baselines, doubling the maximum baseline of the array and improving the array u, v coverage. The improved imaging capabilities will provide an order of magnitude improvement in the noise floor of Murchison Widefield Array continuum images. The upgrade retains all of the features that have underpinned the Murchison Widefield Array's success (large field of view, snapshot image quality, and pointing agility) and boosts the scientific potential with enhanced imaging capabilities and by enabling new calibration strategies

Science with the Murchison Widefield Array : Phase l results and Phase II opportunities

The Murchison Widefield Array (MWA) is an open access telescope dedicated to studying the low-frequency (80-300 MHz) southern sky. Since beginning operations in mid-2013, the MWA has opened a new observational window in the southern hemisphere enabling many science areas. The driving science objectives of the original design were to observe 21 cm radiation from the Epoch of Reionisation (EoR), explore the radio time domain, perform Galactic and extragalactic surveys, and monitor solar, heliospheric, and ionospheric phenomena. All together programs recorded 20 000 h producing 146 papers to date. In 2016, the telescope underwent a major upgrade resulting in alternating compact and extended configurations. Other upgrades, including digital back-ends and a rapid-response triggering system, have been developed since the original array was commissioned. In this paper, we review the major results from the prior operation of the MWA and then discuss the new science paths enabled by the improved capabilities. We group these science opportunities by the four original science themes but also include ideas for directions outside these categories