1 research outputs found
Time series data mining for the Gaia variability analysis
Gaia is an ESA cornerstone mission, which was successfully launched December
2013 and commenced operations in July 2014. Within the Gaia Data Processing and
Analysis consortium, Coordination Unit 7 (CU7) is responsible for the
variability analysis of over a billion celestial sources and nearly 4 billion
associated time series (photometric, spectrophotometric, and spectroscopic),
encoding information in over 800 billion observations during the 5 years of the
mission, resulting in a petabyte scale analytical problem. In this article, we
briefly describe the solutions we developed to address the challenges of time
series variability analysis: from the structure for a distributed data-oriented
scientific collaboration to architectural choices and specific components used.
Our approach is based on Open Source components with a distributed, partitioned
database as the core to handle incrementally: ingestion, distributed
processing, analysis, results and export in a constrained time window.Comment: 4 pages, 3 figures. appears in the Proc. of the 2014 conference on
Big Data from Space (BiDS14), European Commission, Joint Research Centre, P.
Soille, P. G. Marchetti (eds