15,699 research outputs found
Formal Representation of the SS-DB Benchmark and Experimental Evaluation in EXTASCID
Evaluating the performance of scientific data processing systems is a
difficult task considering the plethora of application-specific solutions
available in this landscape and the lack of a generally-accepted benchmark. The
dual structure of scientific data coupled with the complex nature of processing
complicate the evaluation procedure further. SS-DB is the first attempt to
define a general benchmark for complex scientific processing over raw and
derived data. It fails to draw sufficient attention though because of the
ambiguous plain language specification and the extraordinary SciDB results. In
this paper, we remedy the shortcomings of the original SS-DB specification by
providing a formal representation in terms of ArrayQL algebra operators and
ArrayQL/SciQL constructs. These are the first formal representations of the
SS-DB benchmark. Starting from the formal representation, we give a reference
implementation and present benchmark results in EXTASCID, a novel system for
scientific data processing. EXTASCID is complete in providing native support
both for array and relational data and extensible in executing any user code
inside the system by the means of a configurable metaoperator. These features
result in an order of magnitude improvement over SciDB at data loading,
extracting derived data, and operations over derived data.Comment: 32 pages, 3 figure
The Theoretical Astrophysical Observatory: Cloud-Based Mock Galaxy Catalogues
We introduce the Theoretical Astrophysical Observatory (TAO), an online
virtual laboratory that houses mock observations of galaxy survey data. Such
mocks have become an integral part of the modern analysis pipeline. However,
building them requires an expert knowledge of galaxy modelling and simulation
techniques, significant investment in software development, and access to high
performance computing. These requirements make it difficult for a small
research team or individual to quickly build a mock catalogue suited to their
needs. To address this TAO offers access to multiple cosmological simulations
and semi-analytic galaxy formation models from an intuitive and clean web
interface. Results can be funnelled through science modules and sent to a
dedicated supercomputer for further processing and manipulation. These modules
include the ability to (1) construct custom observer light-cones from the
simulation data cubes; (2) generate the stellar emission from star formation
histories, apply dust extinction, and compute absolute and/or apparent
magnitudes; and (3) produce mock images of the sky. All of TAO's features can
be accessed without any programming requirements. The modular nature of TAO
opens it up for further expansion in the future.Comment: 17 pages, 11 figures, 2 tables; accepted for publication in ApJS. The
Theoretical Astrophysical Observatory (TAO) is now open to the public at
https://tao.asvo.org.au/. New simulations, models and tools will be added as
they become available. Contact [email protected] if you have data you
would like to make public through TAO. Feedback and suggestions are very
welcom
- …