9,485 research outputs found
DALiuGE: A Graph Execution Framework for Harnessing the Astronomical Data Deluge
The Data Activated Liu Graph Engine - DALiuGE - is an execution framework for
processing large astronomical datasets at a scale required by the Square
Kilometre Array Phase 1 (SKA1). It includes an interface for expressing complex
data reduction pipelines consisting of both data sets and algorithmic
components and an implementation run-time to execute such pipelines on
distributed resources. By mapping the logical view of a pipeline to its
physical realisation, DALiuGE separates the concerns of multiple stakeholders,
allowing them to collectively optimise large-scale data processing solutions in
a coherent manner. The execution in DALiuGE is data-activated, where each
individual data item autonomously triggers the processing on itself. Such
decentralisation also makes the execution framework very scalable and flexible,
supporting pipeline sizes ranging from less than ten tasks running on a laptop
to tens of millions of concurrent tasks on the second fastest supercomputer in
the world. DALiuGE has been used in production for reducing interferometry data
sets from the Karl E. Jansky Very Large Array and the Mingantu Ultrawide
Spectral Radioheliograph; and is being developed as the execution framework
prototype for the Science Data Processor (SDP) consortium of the Square
Kilometre Array (SKA) telescope. This paper presents a technical overview of
DALiuGE and discusses case studies from the CHILES and MUSER projects that use
DALiuGE to execute production pipelines. In a companion paper, we provide
in-depth analysis of DALiuGE's scalability to very large numbers of tasks on
two supercomputing facilities.Comment: 31 pages, 12 figures, currently under review by Astronomy and
Computin
Radio Frequency Interference Mitigation
Radio astronomy observational facilities are under constant upgradation and
development to achieve better capabilities including increasing the time and
frequency resolutions of the recorded data, and increasing the receiving and
recording bandwidth. As only a limited spectrum resource has been allocated to
radio astronomy by the International Telecommunication Union, this results in
the radio observational instrumentation being inevitably exposed to undesirable
radio frequency interference (RFI) signals which originate mainly from
terrestrial human activity and are becoming stronger with time. RFIs degrade
the quality of astronomical data and even lead to data loss. The impact of RFIs
on scientific outcome is becoming progressively difficult to manage. In this
article, we motivate the requirement for RFI mitigation, and review the RFI
characteristics, mitigation techniques and strategies. Mitigation strategies
adopted at some representative observatories, telescopes and arrays are also
introduced. We also discuss and present advantages and shortcomings of the four
classes of RFI mitigation strategies, applicable at the connected causal
stages: preventive, pre-detection, pre-correlation and post-correlation. The
proper identification and flagging of RFI is key to the reduction of data loss
and improvement in data quality, and is also the ultimate goal of developing
RFI mitigation techniques. This can be achieved through a strategy involving a
combination of the discussed techniques in stages. Recent advances in high
speed digital signal processing and high performance computing allow for
performing RFI excision of large data volumes generated from large telescopes
or arrays in both real time and offline modes, aiding the proposed strategy.Comment: 26 pages, 10 figures, Chinese version accepted for publication in
Acta Astronomica Sinica; English version to appear in Chinese Astronomy and
Astrophysic
Robust sparse image reconstruction of radio interferometric observations with purify
Next-generation radio interferometers, such as the Square Kilometre Array
(SKA), will revolutionise our understanding of the universe through their
unprecedented sensitivity and resolution. However, to realise these goals
significant challenges in image and data processing need to be overcome. The
standard methods in radio interferometry for reconstructing images, such as
CLEAN, have served the community well over the last few decades and have
survived largely because they are pragmatic. However, they produce
reconstructed inter\-ferometric images that are limited in quality and
scalability for big data. In this work we apply and evaluate alternative
interferometric reconstruction methods that make use of state-of-the-art sparse
image reconstruction algorithms motivated by compressive sensing, which have
been implemented in the PURIFY software package. In particular, we implement
and apply the proximal alternating direction method of multipliers (P-ADMM)
algorithm presented in a recent article. First, we assess the impact of the
interpolation kernel used to perform gridding and degridding on sparse image
reconstruction. We find that the Kaiser-Bessel interpolation kernel performs as
well as prolate spheroidal wave functions, while providing a computational
saving and an analytic form. Second, we apply PURIFY to real interferometric
observations from the Very Large Array (VLA) and the Australia Telescope
Compact Array (ATCA) and find images recovered by PURIFY are higher quality
than those recovered by CLEAN. Third, we discuss how PURIFY reconstructions
exhibit additional advantages over those recovered by CLEAN. The latest version
of PURIFY, with developments presented in this work, is made publicly
available.Comment: 22 pages, 10 figures, PURIFY code available at
http://basp-group.github.io/purif
SlicerAstro: a 3-D interactive visual analytics tool for HI data
SKA precursors are capable of detecting hundreds of galaxies in HI in a
single 12 hours pointing. In deeper surveys one will probe more easily faint HI
structures, typically located in the vicinity of galaxies, such as tails,
filaments, and extraplanar gas. The importance of interactive visualization has
proven to be fundamental for the exploration of such data as it helps users to
receive immediate feedback when manipulating the data. We have developed
SlicerAstro, a 3-D interactive viewer with new analysis capabilities, based on
traditional 2-D input/output hardware. These capabilities enhance the data
inspection, allowing faster analysis of complex sources than with traditional
tools. SlicerAstro is an open-source extension of 3DSlicer, a multi-platform
open source software package for visualization and medical image processing.
We demonstrate the capabilities of the current stable binary release of
SlicerAstro, which offers the following features: i) handling of FITS files and
astronomical coordinate systems; ii) coupled 2-D/3-D visualization; iii)
interactive filtering; iv) interactive 3-D masking; v) and interactive 3-D
modeling. In addition, SlicerAstro has been designed with a strong, stable and
modular C++ core, and its classes are also accessible via Python scripting,
allowing great flexibility for user-customized visualization and analysis
tasks.Comment: 18 pages, 11 figures, Accepted by Astronomy and Computing.
SlicerAstro link: https://github.com/Punzo/SlicerAstro/wiki#get-slicerastr
The Data Big Bang and the Expanding Digital Universe: High-Dimensional, Complex and Massive Data Sets in an Inflationary Epoch
Recent and forthcoming advances in instrumentation, and giant new surveys,
are creating astronomical data sets that are not amenable to the methods of
analysis familiar to astronomers. Traditional methods are often inadequate not
merely because of the size in bytes of the data sets, but also because of the
complexity of modern data sets. Mathematical limitations of familiar algorithms
and techniques in dealing with such data sets create a critical need for new
paradigms for the representation, analysis and scientific visualization (as
opposed to illustrative visualization) of heterogeneous, multiresolution data
across application domains. Some of the problems presented by the new data sets
have been addressed by other disciplines such as applied mathematics,
statistics and machine learning and have been utilized by other sciences such
as space-based geosciences. Unfortunately, valuable results pertaining to these
problems are mostly to be found only in publications outside of astronomy. Here
we offer brief overviews of a number of concepts, techniques and developments,
some "old" and some new. These are generally unknown to most of the
astronomical community, but are vital to the analysis and visualization of
complex datasets and images. In order for astronomers to take advantage of the
richness and complexity of the new era of data, and to be able to identify,
adopt, and apply new solutions, the astronomical community needs a certain
degree of awareness and understanding of the new concepts. One of the goals of
this paper is to help bridge the gap between applied mathematics, artificial
intelligence and computer science on the one side and astronomy on the other.Comment: 24 pages, 8 Figures, 1 Table. Accepted for publication: "Advances in
Astronomy, special issue "Robotic Astronomy
Grids and the Virtual Observatory
We consider several projects from astronomy that benefit from the Grid paradigm and
associated technology, many of which involve either massive datasets or the federation
of multiple datasets. We cover image computation (mosaicking, multi-wavelength
images, and synoptic surveys); database computation (representation through XML,
data mining, and visualization); and semantic interoperability (publishing, ontologies,
directories, and service descriptions)
Web-Based Visualization of Very Large Scientific Astronomy Imagery
Visualizing and navigating through large astronomy images from a remote
location with current astronomy display tools can be a frustrating experience
in terms of speed and ergonomics, especially on mobile devices. In this paper,
we present a high performance, versatile and robust client-server system for
remote visualization and analysis of extremely large scientific images.
Applications of this work include survey image quality control, interactive
data query and exploration, citizen science, as well as public outreach. The
proposed software is entirely open source and is designed to be generic and
applicable to a variety of datasets. It provides access to floating point data
at terabyte scales, with the ability to precisely adjust image settings in
real-time. The proposed clients are light-weight, platform-independent web
applications built on standard HTML5 web technologies and compatible with both
touch and mouse-based devices. We put the system to the test and assess the
performance of the system and show that a single server can comfortably handle
more than a hundred simultaneous users accessing full precision 32 bit
astronomy data.Comment: Published in Astronomy & Computing. IIPImage server available from
http://iipimage.sourceforge.net . Visiomatic code and demos available from
http://www.visiomatic.org
The radio structure of 3C 316, a galaxy with double-peaked narrow optical emission lines
The galaxy 3C\,316 is the brightest in the radio band among the
optically-selected candidates exhibiting double-peaked narrow optical emission
lines. Observations with the Very Large Array (VLA), Multi-Element Remotely
Linked Interferometer Network (e-MERLIN), and the European VLBI Network (EVN)
at 5\,GHz have been used to study the radio structure of the source in order to
determine the nature of the nuclear components and to determine the presence of
radio cores. The e-MERLIN image of 3C 316 reveals a collimated coherent
east-west emission structure with a total extent of about 3 kpc. The EVN image
shows seven discrete compact knots on an S-shaped line. However, none of these
knots could be unambiguously identified as an AGN core. The observations
suggest that the majority of the radio structure belongs to a powerful radio
AGN, whose physical size and radio spectrum classify it as a compact
steep-spectrum source. Given the complex radio structure with radio blobs and
knots, the possibility of a kpc-separation dual AGN cannot be excluded if the
secondary is either a naked core or radio quiet.Comment: 12 pages, 3 figures, 2 tables. Accepted for publication in the MNRA
- …