4 research outputs found
DALiuGE: A Graph Execution Framework for Harnessing the Astronomical Data Deluge
The Data Activated Liu Graph Engine - DALiuGE - is an execution framework for
processing large astronomical datasets at a scale required by the Square
Kilometre Array Phase 1 (SKA1). It includes an interface for expressing complex
data reduction pipelines consisting of both data sets and algorithmic
components and an implementation run-time to execute such pipelines on
distributed resources. By mapping the logical view of a pipeline to its
physical realisation, DALiuGE separates the concerns of multiple stakeholders,
allowing them to collectively optimise large-scale data processing solutions in
a coherent manner. The execution in DALiuGE is data-activated, where each
individual data item autonomously triggers the processing on itself. Such
decentralisation also makes the execution framework very scalable and flexible,
supporting pipeline sizes ranging from less than ten tasks running on a laptop
to tens of millions of concurrent tasks on the second fastest supercomputer in
the world. DALiuGE has been used in production for reducing interferometry data
sets from the Karl E. Jansky Very Large Array and the Mingantu Ultrawide
Spectral Radioheliograph; and is being developed as the execution framework
prototype for the Science Data Processor (SDP) consortium of the Square
Kilometre Array (SKA) telescope. This paper presents a technical overview of
DALiuGE and discusses case studies from the CHILES and MUSER projects that use
DALiuGE to execute production pipelines. In a companion paper, we provide
in-depth analysis of DALiuGE's scalability to very large numbers of tasks on
two supercomputing facilities.Comment: 31 pages, 12 figures, currently under review by Astronomy and
Computin
IVOA Recommendation: Simple Spectral Access Protocol Version 1.1
The Simple Spectral Access (SSA) Protocol (SSAP) defines a uniform interface
to remotely discover and access one dimensional spectra. SSA is a member of an
integrated family of data access interfaces altogether comprising the Data
Access Layer (DAL) of the IVOA. SSA is based on a more general data model
capable of describing most tabular spectrophotometric data, including time
series and spectral energy distributions (SEDs) as well as 1-D spectra; however
the scope of the SSA interface as specified in this document is limited to
simple 1-D spectra, including simple aggregations of 1-D spectra. The form of
the SSA interface is simple: clients first query the global resource registry
to find services of interest and then issue a data discovery query to selected
services to determine what relevant data is available from each service; the
candidate datasets available are described uniformly in a VOTable format
document which is returned in response to the query. Finally, the client may
retrieve selected datasets for analysis. Spectrum datasets returned by an SSA
spectrum service may be either precomputed, archival datasets, or they may be
virtual data which is computed on the fly to respond to a client request.
Spectrum datasets may conform to a standard data model defined by SSA, or may
be native spectra with custom project-defined content. Spectra may be returned
in any of a number of standard data formats. Spectral data is generally stored
externally to the VO in a format specific to each spectral data collection;
currently there is no standard way to represent astronomical spectra, and
virtually every project does it differently. Hence spectra may be actively
mediated to the standard SSA-defined data model at access time by the service,
so that client analysis programs do not have to be familiar with the
idiosyncratic details of each data collection to be accessed
VOFilter, Bridging Virtual Observatory and Industrial Office Applications
VOFilter is an XML based filter developed by the Chinese Virtual Observatory
project to transform tabular data files from VOTable format into OpenDocument
format. VOTable is an XML format defined for the exchange of tabular data in
the context of the Virtual Observatory (VO). It is the first Proposed
Recommendation defined by International Virtual Observatory Alliance, and has
obtained wide support from both the VO community and many Astronomy projects.
OpenOffice.org is a mature, open source, front office applications suite with
the advantage of native support of industrial standard OpenDocument XML file
format. Using the VOFilter, VOTable files can be loaded in OpenOffice.org Calc,
a spreadsheet application, and then displayed and analyzed as other spreadsheet
files. Here, the VOFilter acts as a connector, bridging the coming VO with
current industrial office applications. Virtual Observatory and technical
background of the VOFilter are introduced. Its workflow, installation and usage
are presented. Existing problems and limitations are also discussed together
with the future development plans.Comment: Accepted for publication in ChJAA (9 pages, 2 figures, 185KB
IVOA Recommendation: Spectrum Data Model 1.1
We present a data model describing the structure of spectrophotometric
datasets with spectral and temporal coordinates and associated metadata. This
data model may be used to represent spectra, time series data, segments of SED
(Spectral Energy Distributions) and other spectral or temporal associations.Comment: http://www.ivoa.ne