This paper describes a solution found during recent research that could provide improvements in the efficiency, reliability and cost of retrieving stored astronomical data. This solution uses XML Technologies in showing that when querying a variety of astronomical data sources a standardised data structure can be output into an XML query results Document. This paper shows the astronomical XMLSchema that has been partially developed in conjunction with simple custom supporting system software. It also discusses briefly possible future implications

Altieri, Bruno

Belikov, Andrey

Boxhoorn, Danny

Buenadicha, Guillermo

de Teodoro, Pilar

Droge, Bob

Marchetti, Pier Giorgio

McFarland, John

Nieto, Sara

Salgado, Jesus

Soille, Pierre

Tsyganov, Andrey

Valentijn, E

Williams, Owen

English

arXiv

The Euclid Archive System (EAS) is in the core of the Euclid Science Ground Segment (SGS). It supports the processing and storage of Euclid data from the input raw frames to the science-ready images and catalogs. We review the architectural design of the system, implementation progress and main challenges in the building of the EAS

NARCIS 

The Euclid Archive System: A Datacentric Approach to Big Data

International audienceThe paper explores how multimedia approaches used in image understanding tasks could be adapted for remote sensing image analysis. In a first step, we show on 3 channels color images through the UC Merced Land Use Dataset how Deep Learning approach provides a significant performance increase compared to Bag of VisualWords approach. In a second step, we propose an extension of deep learning scheme to deal with hyperspectral data. The proposed scheme is based on a 3D architecture which jointly processes spectral and spatial information

Ben Hamida, Amina

Benoit, A

Lambert, Patrick

Ben-Amar, Chokri

Hal - Université Grenoble Alpes

DEEP LEARNING APPROACH FOR REMOTE SENSING IMAGE ANALYSIS

The Copernicus programme of the European Union with its numerous Sentinel satellites operated by the European Space Agency are effectively making Earth Observation (EO) entering the big data era.  This means that most current application projects cannot be addressed with conventional techniques.  That is, the EO data revolution brough by Copernicus needs to be matched by a processing revolution.  Existing approaches such as those based on the processing of massive archives of Landsat data are reviewed and the concept of the Joint Research Centre Earth Observation Data and Processing platform is briefly presented.JRC.E.1-Disaster Risk Managemen

SOILLE PIERRE

BURGER ARMIN

RODRIGUEZ ASERETTO ROQUE DARIO

SYRRIS VASILEIOS

VASILEV VESELIN

JRC Publications Repository

TOWARDS A JRC EARTH OBSERVATION DATA AND PROCESSING PLATFORM
P. Soille1, A. Burger2, D. Rodriguez1, V. Syrris1, and V. Vasilev2
European Commission, Joint Research Centre (JRC)
1Institute for the Protection and Security of the Citizens, Global Security and Crisis Management Unit
2Institute for Environment and Sustainability, Digital Earth and Reference Data Unit
ABSTRACT
The Copernicus programme of the European Union with its
fleet of Sentinel satellites operated by the European Space
Agency are effectively making Earth Observation (EO) enter-
ing the big data era. Consequently, most application projects
at continental or global scale cannot be addressed with con-
ventional techniques. That is, the EO data revolution brought
in by Copernicus needs to be matched by a processing revolu-
tion. Existing approaches such as those based on the process-
ing of massive archives of Landsat data are reviewed and the
concept of the Joint Research Centre Earth Observation Data
and Processing platform is briefly presented.
Index Terms— Earth Observation, Sentinel, Copernicus,
Infrastructure
1. INTRODUCTION
To date, the United States (U.S.) Government is the largest
provider of environmental and Earth system data in the
world1. A first data revolution happened in 2008 when the
U.S. Geological Survey decided to release for free to the pub-
lic its Landsat archive which is the worlds largest collection
of Earth imagery [11]. Still, the European Commission, with
its ambitious Copernicus programme and associated Sentinel
missions (S1 to S6 satellite series) operated by the European
Space Agency and complemented by a range of contribut-
ing missions, is on the way to become the main provider of
global EO data with a free, full, and open access data policy.
With expected data volumes of 10 TB per day (when all Sen-
tinel series will reach full operational capacity), data velocity
highlighted by the production of global coverage with repeat
time as short as 2 days for Sentinel-3, and data variety result-
ing from sensors in the optical and radar ranges at various
spatial, spectral, and temporal resolutions, the Copernicus
programme is a game changer making EO data effectively
entering the big data era [10]. Figure 1 shows the overall
estimated data throughput for the Sentinel 1–3 missions com-
pared to those delivered by the Landsat 8/MODIS satellites.
1http://dels.nas.edu/resources/static-assets/besr/
miscellaneous/Stryker.pdf
Fig. 1. Yearly data flow estimates from Sentinel 1–3 (as-
suming full operational capacity) compared to MODIS and
Landsat 8 data flows.
Whilst the European Union (EU) is making EO entering
the big data era in terms of data production, innovative de-
velopments need to be pursued to fully exploit the potential
of the generated data whether for academic, institutional, or
commercial applications. This also applies to the Joint Re-
search Centre where the current fragmented approach of EO
data storage and processing is no longer sustainable.
2. THE SENTINELS AND THE BIG EO DATA ERA
The evolution of the cumulative data produced by the Land-
sat missions and Sentinel 1-2-3 with estimations until sum-
mer 2018 is shown in Fig. 2. The underlying calculations are
based on the following assumptions and considerations:
1. The data volume of Landsat 1-6 missions is ∼120 TB2;
2. The data volume of Landsat 7 and 8 is estimated from
the relating metadata files provided by USGS3;
3. MODIS Terra and Aqua generate 70GB/day each [7];
2http://academic.emporia.edu/aberjame/remote/
landsat/landsat.htm
3http://landsat.usgs.gov/metadatalist.php
Exploitation Platforms 2
Proc. of the 2016 conference on
Big Data from Space (BiDS’16) doi: 10.2788/854791
65 Santa Cruz de Tenerife, Spain
15–17 March 2016
Fig. 2. Data volume: evolution of the data produced by the MODIS/Landsat missions and Sentinel 1-2-3 (estimations based
on data throughput in full operational capacity after six months of the scheduled launch dates).
Fig. 3. Data velocity of Sentinel 1–3: daily data throughput
in full operational capacity and orbit repeat time (in days).
4. The data volume of Sentinel 1, 2 and 3 in their twin con-
stellations is approximately 3.6 TB/day, 1.6 TB/day and
0.6 TB/day respectively4; all starting dates are consid-
ered to be around six months after the respective mis-
sion launch (assuming that data are generated in full
operational capacity by that date).
These estimations indicate that the cumulated data generated
by the Sentinel 1–3 missions will exceed the data generated
by all Landsat missions during 2018.
The data velocity daily of S1–3/A-B is highlighted by
their data throughput and orbit repeat times, see Fig. 3. Fi-
nally, data variety of the Sentinel 1 to 3 satellite series is sum-
marised in Table 1. Revisit times are not indicated since they
increase with latitude.
4https://www.ffg.at/sites/default/files/esa_
sentinel_core_products_overview.pdf
3. PLATFORMS FOR BIG EO DATA
This section presents a brief survey of some existing plat-
forms and other initiatives to address the needs of big EO
data. Given the available space for this short paper, it is by
no means comprehensive. In particular, platforms mostly de-
voted to data dissemination are not considered here.
3.1. Examples from public sector
• NASA Earth Exchange (NEX)5 is a platform for scien-
tific collaboration, knowledge sharing and research in
the Earth science community;
• ESA Exploitation Platforms (EP): Thematic Exploita-
tion Platforms (TEPs), Earth Exploitation Platforms
(EOPs) and Sentinel Application Platform (SNAP);
• DLR GeoFarm hardware organisation follows the
cloud-like vitalisation of processing hardware, see
also [4].
• The Theia Land Data Centre is a French national
inter-agency organisation designed to foster the use
of images issued from the space observation of land
surfaces [6];
• Earth Observation Data Center (EODC)6 public/private
initiative [2] that combines HPC with data provision
and collaborative development;
• Australian Geoscience Data Cube (AGDC) at Na-
tional Computational Infrastructure in Canberra, Aus-
tralia [5].
5https://nex.nasa.gov
6https://www.eodc.eu
Exploitation Platforms 2
Proc. of the 2016 conference on
Big Data from Space (BiDS’16) doi: 10.2788/854791
66 Santa Cruz de Tenerife, Spain
15–17 March 2016
Table 1. Sentinel 1 to 3 data variety: spectral, temporal, and spatial (repeat cycle/global coverage for 1 satellite).
Mission Sensors Applications Repeat cycle/
Global coverage
Resolution Formats
Sentinel-1 C-Band SAR Monitoring: sea ice, oil spills, marine
winds and waves, land-use change, re-
spond emergencies such as floods and
earthquakes
12 days/12 days Strip map mode 80km
swath 5x5m; Interfero-
metric wide swath 240km
5x20m; Extra wide swath
400km 25x100m; Wave
mode 20x20km at 5x20m
.SAFE with GEO-
TIFF, XML, PNG,
XSD, HTML and
netCDF files
Sentinel-2 MSI (13 bands from
443 nm to 2,190 nm)
Monitoring agriculture, forests, land-
use change, land-cover change; map-
ping biophysical variables; monitoring
coastal and inland waters; risk map-
ping and disaster mapping
10 days/10 days 10m, 20m, and 60m spa-
tial resolution to identify
spatial details consistent
with 1ha minimum map-
ping unit)
.SAFE with
JPG2000, XML,
GML and HTML
files
Sentinel-3 OLCI (21 bands from
400 nm to 1,020 nm),
SLSTR (9 bands
from 555 nm to
10,850 nm), SRAL,
MWR
Monitoring sea (ocean circulation,
tides), coastal zone, inland waters and
land, ice and sea-ice, climate geodesy
and geophysics, and land topography
27 days/3 days OLCI 300m, SLSTR
(500m for solar re-
flectance, 1km for thermal
infrared bands)
.SAFE with GEO-
TIFF, XML, PNG,
XSD, HTML and
netCDF files
3.2. Examples from private sector
Several companies are hosting large amounts of EO data in
combination with processing capabilities, e.g., Google Earth
Engine (GEE)7 for a web-based platform with dedicated
web API, Amazon Web Services (AWS) with availability of
Sentinel-28 and Landsat-89, and CloudEO10 that proposes a
geo-infrastructure as a service.
3.3. Big EO data initiatives
In parallel to the development of public and private platforms,
a number of governmental and research initiatives aiming at
addressing the needs of big geospatial data are flourishing;
see for example:
• BEDI is a U.S.A. government big data initiative on civil
Earth observation11;
• The Big SkyEarth EU COST action12;
• The EarthServer: http://www.earthserver.eu/ see also [1];
• EarthCube is a joint initiative between the Division
of Advanced Cyberinfrastructure (ACI) and the Geo-
sciences Directorate (GEO) of the US National Science
Foundation (NSF).
7https://earthengine.google.com
8http://sentinel-pds.s3-website.eu-central-1.
amazonaws.com
9http://aws.amazon.com/public-data-sets/landsat
10http://www.cloudeo-ag.com
11http://www.whitehouse.gov/sites/default/files/
microsites/ostp/NSTC/national_plan_for_civil_
earth_observations_-_july_2014.pdf
12http://www.cost.eu/COST_Actions/TDP/Actions/
TD1403
• The Earth System Grid Federation: An Open Infras-
tructure for Access to Distributed Geospatial Data;
• NOAA’s Big Data Project within which a set of Data
Alliances are being formed with providers of IaaS.
4. JRC EO DATA AND PROCESSING PLATFORM
A number of JRC projects are exploiting Earth Observation
data to achieve their goals. In the Sentinel era, this can
only be addressed by an integrated approach combining data
storage and data processing. This leads to the concept of
JRC EO Data and Processing Platform (JEODPP) developed
in the framework of the JRC EO&Social Sensing Big Data
(EO&SS@BD) pilot project.
The envisaged architecture consists of processing servers
accessing the data provided by a series of storage servers
and their directly attached storage (Just a Bunch of Disks or
JBODs) in a distributed file system environment. The I/O
bottleneck typically observed with network attached storage
is avoided by considering appropriate high speed server inter-
communication topology (switched fabric in fibre channel).
This topology has the best scalability compared to arbitrated
loop and point-to-point alternatives. Storage servers are au-
tomatically populated with the data requested by the applica-
tions. For example, the automatic download of Sentinel-2A
data is achieved by using a time-based job scheduler launch-
ing OpenSearch and OpenData (ODat) scripts taking into ac-
count user requirements (geographical areas, cloud coverage,
seasonality, etc.).
Processing can be performed at various levels through a
sandbox environment. In its simplest utilisation, users have
direct access to a batch job scheduler allowing the automatic
and distributed processing of EO data at continental or global
scale. For example, the detection of clouds on the full S2A
Exploitation Platforms 2
Proc. of the 2016 conference on
Big Data from Space (BiDS’16) doi: 10.2788/854791
67 Santa Cruz de Tenerife, Spain
15–17 March 2016
Fig. 4. Components of the planned JRC EO data and pro-
cessing platform (JEODPP).
expert user data (about 30 TB of JPEG2000 compressed im-
agery) in only 5 days on a computing cluster with 10 process-
ing nodes. Similarly, the automatic mosaicing of 2.5m SPOT
(Copernicus CORE 3 data set) covering the whole territory
of the EU plus 11 additional states (European Environment
Agency member and associate members) [9] as well as the
computation of Global Human Settlement Layers [8] from 4
multitemporal global Landsat data sets are achieved within
one week.
The sandbox will offer different levels of user interaction
with the platform:
• direct access to the batch job scheduler (slurm or HT-
Condor);
• virtualisation/operating-system-level virtualisation of
the desired environment for prototyping (based on
Docker Linux containers);
• interactive EO visualisation/processing capabilities;
• interactive data science and scientific computing through
Jupyter web application (IPython notebooks), see
also [3].
The various components of the planned EO data and process-
ing platform are sketched in Figure 4.
5. CONCLUDING REMARKS
The EU Copernicus programme with its series of Sentinel
missions acts as a game changer by bringing EO into the big
data era. The value of the produced data depends on our ca-
pacity to extract information from it. The velocity, variety,
and volume of the generated data combined with the need
for using other non EO data sources calls for innovative ap-
proaches in data storage and processing. The scope of the JRC
EO&Social Sensing Big Data (EO&SS@BD) pilot project is
to propose innovative solutions addressing the needs of JRC
projects. We are currently testing and optimising the various
components of the JEODPP to ensure its scalability.
6. REFERENCES
[1] P. Baumann et al. Big data analytics for earth sciences:
the earthserver approach. International Journal of Dig-
ital Earth, 1–27, 2015. doi: 10.1080/17538947.
2014.1003106.
[2] C. Briese, et al. Challenges in the exploitation of big
earth observation data. In Proc. of BiDS’14, 49–52.
2014. doi: 10.2788/1823.
[3] A. Burger et al. Towards an infrastucture for inter-
active Earth Observation data analysis and processing.
In B. Chan, editor, Cloud Services for Synchronisation
and Sharing, p. 27. ETHZ, 2016. doi: 10.5281/
zenodo.44783.
[4] S. Kiemle, et al. Big data management in earth observa-
tion - the German satellite data archive at DLR. In Proc.
of BiDS’14, pages 45–49. doi: 10.2788/1823.
[5] A. Lewis, et al. Iterating Petabyte-Scale Earth Observa-
tion Processes in The Australian Geoscience Data Cube.
In Proc. of BiDS’14. doi: 10.2788/1823.
[6] C. L’Helguen et al. Muscate: Multi-satellites, multi-
sensors and multi-temporal theia data centre. In Proc. of
BiDS’14, 114–118. doi: 10.2788/1823.
[7] E.J. Masuoka, et al. Key characteristics of MODIS data
products. IEEE Transactions on Geoscience and Remote
Sensing, 36(4):1313–1323, 1998. doi: 10.1109/36.
701081.
[8] M. Pesaresi, et al. Operating procedure for the produc-
tion of the global human settlement layer from Landsat
data of the epochs 1975, 1990, 2000, and 2014. Techni-
cal Report, JRC, 2015. doi: 10.2788/253582.
[9] P. Soille. Seamless mosaicing of very high resolution
satellite data at continental scale. In P. Soille et al.,
editors, Proc. of BiDS’14, 222–223. doi: 10.2788/
1823.
[10] P. Soille et al. Preface. In Proc. of the BiDS’14 doi:
10.2788/1823.
[11] The Landsat Science Team. Free access to Landsat im-
agery. Science, 320(5879):1011, 2008. doi: 10.1126/
science.320.5879.1011a.
Exploitation Platforms 2
Proc. of the 2016 conference on
Big Data from Space (BiDS’16) doi: 10.2788/854791
68 Santa Cruz de Tenerife, Spain
15–17 March 2016


Towards a JRC Earth Observation Data and Processing Platform

HAL Université de Savoie

Dissertations of the University of Groningen

   University of GroningenThe Euclid Archive System: A Datacentric Approach to Big DataBelikov, Andrey; Williams, Owen; Altieri, Bruno; Boxhoorn, Danny; Buenadicha, Guillermo ;Droge, Bob; McFarland, John; Nieto, Sara; Salgado, Jesus; de Teodoro, PilarPublished in:Proceedings of 2016 conference on Big Data from Space (BiDS'16)DOI:10.2788/854791IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionEarly version, also known as pre-printPublication date:2016Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Belikov, A., Williams, O., Altieri, B., Boxhoorn, D., Buenadicha, G., Droge, B., McFarland, J., Nieto, S.,Salgado, J., de Teodoro, P., Tsyganov, A., & Valentijn, E. (2016). The Euclid Archive System: ADatacentric Approach to Big Data. In P. Soille, & P. G. Marchetti (Eds.), Proceedings of 2016 conferenceon Big Data from Space (BiDS'16) (Vol. JRC100655, pp. 212-215). Publications Office of the EuropeanUnion . https://doi.org/10.2788/854791CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 09-09-2023THE EUCLID ARCHIVE SYSTEM:   A  DATA-CENTRIC APPROACH TO BIG DATA A.N.Belikov (1), O.R.Williams(2), B.Altieri(3), D.Boxhoorn(1) , G.Buenadicha(3), B. Droege(2),   J.McFarland(1), S.Nieto(3), J. Salgado(3), P. de Teodoro(3), A.Tsyganov(2), E.A. Valentijn(1) (1) Kapteyn Astonomical Institute, University of Groningen, Groningen, The Netherlands (2) Donald Smits Centre for Information Technology, University of Groningen, Groningen, The Netherlands (3) European Space Astronomy Center, European Space Agency, Spain  ABSTRACT The Euclid Archive System (EAS) is in the core of the Euclid Science Ground Segment (SGS). It supports   the processing and storage of  Euclid data from the input raw frames to the science-ready images and catalogs. We review the architectural design of the system, implementation progress and main challenges in the building of  the EAS.   Index Terms— Euclid, Data Storage, Information System 1. INTRODUCTION Euclid is an ESA M2 mission and a milestone in the understanding of the geometry of the Universe [1]. Euclid faces two  main challenges from the point of view of the data processing.  Firstly, the unprecedented accuracy which must be achieved in order to meet the scientific goals. Secondly, the mission will depend heavily on the processing and reprocessing of the ground-based data which will form the bulk of the data volume to store [2]. In total Euclid will produce up to 26 PB per year of observations [3]. 2.  EAS AND SGS The Euclid Science Ground Segment is a distributed data processing and data storage system which is responsible for the delivery of the science-ready data to ESA [3]. The SGS is formed by 9 national Science Data Centres and the EuclidScience Operations Centre. Each data centre provides resources for processing, along with expertize in the coding of Euclid pipelines.  The Euclid Archive System (EAS) is a core element of the SGS implementing data-centric approach [4] to the data processing.  In the SGS and the EAS the data is a combination of data files (images, spectra, catalogs) and metadata. The metadata includes the full  data lineage, which is necessary to achive the goals of the Euclid mission.   The EAS must provide to SGS users and subsystems of SGS the ability to trace any bit of information produced in the SGS and must  provide quality information on the produced images, spectra and catalogs. According to our experience in previous missions (Astro-WISE [4] and LOFAR Long-Term Archive [5])  the data volume of metadata will not exceed 5% of the total data volume. The clear difference between the bulk of the data stored in files and extensive metadata describing data products in files allows to move most of data mining operations to the relational database without access to the data files.      The EAS is a joint development between ESA and Euclid Consortium led by the SDC of the Netherlands and the ESDC (ESAC Science Data Centre) . It should be stressed that the EAS is not an archive in the conventional sense. Instead, the EAS is a distributed scientific information system which ensures that any operation with the data is registered and can be traced back to the user and  pipeline.  EAS ensures that SGS can access and process hundreds of Petabytes of data.  3.  EAS DESIGN  Figure 1: Overview of EAS and it's place in Euclid SGS The EAS design is based on requirements, formed by the ESA and the Euclid community, on the data processing in distributed environment. To allow the handling of  numerous and often contradictory requirements on the EAS the whole system was divided on 3 independent parts:  The Data Processing System (DPS) for the interactions within the SGS  The Science Archive System (SAS) for the delivery of data releases, their long-term preservation and support of Euclid scientific use-cases,   The distributed Storage System (DSS) for the storage of data files. Figure 1 shows overview of the EAS design with primary users and their interaction with DPS, SAS and DSS.  The primary task of the DPS is to server as an information system for the production of the data release. To achieve this goal DPS  keeps data lineage for each data object tracing any operation in the system to allow data reproduction and reprocessing. DPS allows to trace each frame, catalog or source catalog back to the original raw data, processing parameters used and pipeline which produced it. As well it allows to see the current status of the processing in SGS and quality of the raw and processed data.   The DSS serves both the  DPS and SAS as a common distributed file storage solution.  In this solution we utilize a distributed approach for the data files storage and a centralized approach to  the storage of the metadata.  The DSS consists of DSS servers which are interfacing the non-homogeneous data storage solutions provided by the national SDCs. At least one DSS server is installed at each SDC and stores the data files processed or created during running of a pipeline in this SDC. The design of the processing plan for each pipeline allows to minimize data file transfer between SDCs. To ensure zero loss of the data at least 2 copies of each data file is created in DSS. Each copy is registered in the metadata storage.  The EAS implements the Euclid Common Data Model (ECDM) which describes both scientific data (data products generated by pipelines) and processing and operational metadata (processing and data distribution orders, location of the file at DSS, processing plan). The ECDM is implemented ``as it is'' inside the DPS and transformed to Science Exploration Data Model for the implementation in SAS.  The interface between SAS and DPS will allow to SAS to retrieve the science metadata from the DPS for the Euclid data release and put it in long-term storage in SAS. The same ECDM is used to bind pipeline data flows.  The SAS is part of the EAS and it's aimed to support the needs for scientific data exploration for Euclid Consortium and wider  astronomical community. The SAS, as part of the Euclid Archive, will face the challenge of Big Data, as it will store a huge and increasing amount of scientific metadata, and catalogues of 10 billion of galaxies. This will provide the worldwide astronomical community with an extremely large source of targets for future missions. Under these premises, the SAS will face the challenge to guarantee the preservation and public access of the data stored to the scientific community.  The SAS is being built at the ESAC Science Data Centre (ESDC), that is in charge of the development and operations of the scientific archives for the ESA Astronomy, Planetary and Heliosphysics missions. The SAS is focused on the needs from the scientific community. In this context, the SAS will provide access to the most valuable scientific metadata coming from the EAS-DPS through a set of public releases. According to the policy of public releases defined by the Euclid Consortium, the plan is to deliver 3 data releases to the scientific community every 2 years after the nominal start of the mission [6].  The design of the EAS is fully hardware independent and partially software independent. Metadata storage is based on a RDBMS, currently Oracle 11g, but it will be possible to switch to other RDBMS in the future. Metadata storage in the DPS and the SAS are completely independent and can be based on different solutions. DSS server can use  local or NFS-mounted filesysytem, Grid SE, sftp server, iRODS, Astro-WISE dataserver and can be extended for the existing or upcoming storage cases.  The ability to extend DSS storage practicaly to any storage solution was introduced in the design to cope with the possible changes during the lifetime of the project and to support the SGS for at least 8 years after the launch.  The design of the SAS follows the latest generation of archives being developed by the ESDC, taking full advantage of the existing knowledge, expertise and code. The SAS builds on top of the latest ESDC’s common Archives Building System Infrastructure (ABSI), which defines the common components to the latest ESA Science Archives (i.e. Gaia) within a three-tier modular architecture (client, server and data layer).  Figure 2: SAS Architecture Diagram 4.  EAS INTERFACES There are a number of interfaces and services which are implemented for EAS. EAS DPS will have to support interaction with SDC for the data processing (through Infrastructure Abstract Layer- IAL). IAL queries EAS DPS for commands for the data distribution and data processing and ingest metadata for newly created data products. The basic metadata browsing service allows to browse content of the metadata storage (according to the visibility scope for each user) and retrieve metadata for the data product in XML or other format.  DPS will be supplied with at least 2 other services – for the manual inspection of the quality of data products and for the retrieval and update of calibration frames.   The SAS will provide two ways of access through the Archive User Services (AUS), a web-based portal and a command line interface for programming access. The web-portal is based on the Google Web Toolkit technology [7] and it has been designed to be easy to use and to provide a friendly interaction.   Figure 3: SAS Web User Interface On the server side, the components are based on Java and it  will integrate standard VO protocols to manage requests from the users. At database level, the metadata repository will support the Science Exploitation Data Model (SEDM), that describes the metadata and catalogues oriented to scientific exploration. At the time of writing, the RDBMS is PostgreSQL join with pgSphere and Q3C modules that provides spherical data types, functions and operators to PostgreSQL.  Finally, the Metadata Transfer Service (MTS) will ingest the science metadata of the data release compliant with the SEDM from the ECDM at DPS, according to the policy of public releases defined by the Euclid Consortium.  The scientific requirements on the SAS mainly covers three different areas: parametric search for metadata and catalogues, data retrieval and visualization of images, spectra, etc. Regarding the maps visualization, it will be based on the technology developed for the ESA Sky [8], that allows the exploration of the astronomical resources using a useful and intuitive web interface. In addition, it will provide a set of tools to allow on-line research.  For the data retrieval, SAS stores the links pointing to the data products as part of its metadata. That links are managed by the DSS servers at the SDCs to deliver the data to the end users. VO protocols play an important role within the architecture of the SAS, but also supporting added value tools integrated into the archive. Table Access Protocol (TAP+) [9] that provides efficient parametric search, has been developed for the Gaia Archive (Reference to Gaia BIDS Contribution), is also part of the SAS infrastructure and can be accessed from the Web interface.    Figure 4: TAP Web Interface in SAS VOSpace [10], the IVOA protocol for distributed data storage, is also integrated as part of the SAS infrastructure as an added value tool for the archive. It provides a storage abstraction layer and sharing capabilities transparent for the user.   Figure 5: VOSpace Web User Interface Other VO protocols like SAMP, will provide interoperability with astronomical analysis applications like Topcat among others.  To conclude, SAS will provide the tools and VO interfaces to enable the Software-to-Data Paradigm and "bring the software to the data" for the Euclid science.  4.  EAS STATUS The EAS Prototype was developed and tested in 2013 and 2014,. In 2015 the prototype formed the basis of the first version of EAS itself.  First interfaces for the DPS and the DSS were released and tested during IT challenges organized by Euclid Consortium producing and storing simulated Euclid data. We have successfully tested massive metadata ingestion for the data objects  with extensive data lineage (KIDS DR1) [11]. In 2015 EAS team tested 2-copies configuration of EAS DPS  metadata storage and metadata transfer between EAS DPS  metadata storage mirrors in ESAC and Groningen.  5. FUTURE DEVELOPMENT In the following years prior to the start of the mission the EAS will go through a number of crucial steps in the development including the final selection of an RDBMS. We plan as well to create a system which will support a dynamic data model:  accommodating  changes in ECDM without re-creation of the metadata storage scheme and migrating the data between different versions  of the data model. Current DPS metadata storage will be separated on DPS metadata storage proper and DSS metadata storage. 6. REFERENCES [1] R. Laureijs et al., Euclid Definition Study Report, arxiv1110.3193, 2011 [2] O.R.Williams et al., Data transmission, handling and dissemination of Euclid, Proceedings of NETSPACE 2014, 11, 2014 [3]  F. Pasian et al., Development Plans for the Euclid Science Ground Segment, Proceedings of ADASS XXIII, ASP Conf. Ser. 485, 505, 2014 [4]   K.Begeman et al., The Astro-WISE datacentric information system, Experimental Astronomy 35, 1, 2013   [5]  K.Begeman et al., LOFAR Information System, Future Generation Computer Systems 27, 319, 2011  [6] Maurice P. et al., Euclid : Big Data From Dark Space, Big Data Spain, Oct. 2015 [7] Dewsbury, R., Google Web Toolkit Applications, ISBN 978-0-321-50196-7, Prentice Hall, (2007). [8] Salgado, J., ESA Sky: A Tool to Discover Astronomical Data from ESA Missions, Big Data From Space 2016.  [9] Dowler P. et at., IVOA Recommendation: Table Access Protocol Version 1.0, 2011arXiv1110.0497D [10] Graham M. et al., IVO Working Draft : VOSpace Version 2.1, WD-VOSpace-2.1-20150601 [11] J.T.A. De Jong et al., Kilo Degree Survey, Experimental Astronomy 35, 24, 2013   

University of Groningen

   University of GroningenThe Euclid Archive System: A Datacentric Approach to Big DataBelikov, Andrey; Williams, Owen; Altieri, Bruno; Boxhoorn, Danny; Buenadicha, Guillermo ;Droge, Bob; McFarland, John; Nieto, Sara; Salgado, Jesus; de Teodoro, PilarPublished in:Proceedings of 2016 conference on Big Data from Space (BiDS'16)DOI:10.2788/854791IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionEarly version, also known as pre-printPublication date:2016Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Belikov, A., Williams, O., Altieri, B., Boxhoorn, D., Buenadicha, G., Droge, B., ... Valentijn, E. (2016). TheEuclid Archive System: A Datacentric Approach to Big Data. In P. Soille, & P. G. Marchetti (Eds.),Proceedings of 2016 conference on Big Data from Space (BiDS'16) (Vol. JRC100655, pp. 212-215).Publications Office of the European Union . https://doi.org/10.2788/854791CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 13-11-2019THE EUCLID ARCHIVE SYSTEM:   A  DATA-CENTRIC APPROACH TO BIG DATA A.N.Belikov (1), O.R.Williams(2), B.Altieri(3), D.Boxhoorn(1) , G.Buenadicha(3), B. Droege(2),   J.McFarland(1), S.Nieto(3), J. Salgado(3), P. de Teodoro(3), A.Tsyganov(2), E.A. Valentijn(1) (1) Kapteyn Astonomical Institute, University of Groningen, Groningen, The Netherlands (2) Donald Smits Centre for Information Technology, University of Groningen, Groningen, The Netherlands (3) European Space Astronomy Center, European Space Agency, Spain  ABSTRACT The Euclid Archive System (EAS) is in the core of the Euclid Science Ground Segment (SGS). It supports   the processing and storage of  Euclid data from the input raw frames to the science-ready images and catalogs. We review the architectural design of the system, implementation progress and main challenges in the building of  the EAS.   Index Terms— Euclid, Data Storage, Information System 1. INTRODUCTION Euclid is an ESA M2 mission and a milestone in the understanding of the geometry of the Universe [1]. Euclid faces two  main challenges from the point of view of the data processing.  Firstly, the unprecedented accuracy which must be achieved in order to meet the scientific goals. Secondly, the mission will depend heavily on the processing and reprocessing of the ground-based data which will form the bulk of the data volume to store [2]. In total Euclid will produce up to 26 PB per year of observations [3]. 2.  EAS AND SGS The Euclid Science Ground Segment is a distributed data processing and data storage system which is responsible for the delivery of the science-ready data to ESA [3]. The SGS is formed by 9 national Science Data Centres and the EuclidScience Operations Centre. Each data centre provides resources for processing, along with expertize in the coding of Euclid pipelines.  The Euclid Archive System (EAS) is a core element of the SGS implementing data-centric approach [4] to the data processing.  In the SGS and the EAS the data is a combination of data files (images, spectra, catalogs) and metadata. The metadata includes the full  data lineage, which is necessary to achive the goals of the Euclid mission.   The EAS must provide to SGS users and subsystems of SGS the ability to trace any bit of information produced in the SGS and must  provide quality information on the produced images, spectra and catalogs. According to our experience in previous missions (Astro-WISE [4] and LOFAR Long-Term Archive [5])  the data volume of metadata will not exceed 5% of the total data volume. The clear difference between the bulk of the data stored in files and extensive metadata describing data products in files allows to move most of data mining operations to the relational database without access to the data files.      The EAS is a joint development between ESA and Euclid Consortium led by the SDC of the Netherlands and the ESDC (ESAC Science Data Centre) . It should be stressed that the EAS is not an archive in the conventional sense. Instead, the EAS is a distributed scientific information system which ensures that any operation with the data is registered and can be traced back to the user and  pipeline.  EAS ensures that SGS can access and process hundreds of Petabytes of data.  3.  EAS DESIGN  Figure 1: Overview of EAS and it's place in Euclid SGS The EAS design is based on requirements, formed by the ESA and the Euclid community, on the data processing in distributed environment. To allow the handling of  numerous and often contradictory requirements on the EAS the whole system was divided on 3 independent parts:  The Data Processing System (DPS) for the interactions within the SGS  The Science Archive System (SAS) for the delivery of data releases, their long-term preservation and support of Euclid scientific use-cases,   The distributed Storage System (DSS) for the storage of data files. Figure 1 shows overview of the EAS design with primary users and their interaction with DPS, SAS and DSS.  The primary task of the DPS is to server as an information system for the production of the data release. To achieve this goal DPS  keeps data lineage for each data object tracing any operation in the system to allow data reproduction and reprocessing. DPS allows to trace each frame, catalog or source catalog back to the original raw data, processing parameters used and pipeline which produced it. As well it allows to see the current status of the processing in SGS and quality of the raw and processed data.   The DSS serves both the  DPS and SAS as a common distributed file storage solution.  In this solution we utilize a distributed approach for the data files storage and a centralized approach to  the storage of the metadata.  The DSS consists of DSS servers which are interfacing the non-homogeneous data storage solutions provided by the national SDCs. At least one DSS server is installed at each SDC and stores the data files processed or created during running of a pipeline in this SDC. The design of the processing plan for each pipeline allows to minimize data file transfer between SDCs. To ensure zero loss of the data at least 2 copies of each data file is created in DSS. Each copy is registered in the metadata storage.  The EAS implements the Euclid Common Data Model (ECDM) which describes both scientific data (data products generated by pipelines) and processing and operational metadata (processing and data distribution orders, location of the file at DSS, processing plan). The ECDM is implemented ``as it is'' inside the DPS and transformed to Science Exploration Data Model for the implementation in SAS.  The interface between SAS and DPS will allow to SAS to retrieve the science metadata from the DPS for the Euclid data release and put it in long-term storage in SAS. The same ECDM is used to bind pipeline data flows.  The SAS is part of the EAS and it's aimed to support the needs for scientific data exploration for Euclid Consortium and wider  astronomical community. The SAS, as part of the Euclid Archive, will face the challenge of Big Data, as it will store a huge and increasing amount of scientific metadata, and catalogues of 10 billion of galaxies. This will provide the worldwide astronomical community with an extremely large source of targets for future missions. Under these premises, the SAS will face the challenge to guarantee the preservation and public access of the data stored to the scientific community.  The SAS is being built at the ESAC Science Data Centre (ESDC), that is in charge of the development and operations of the scientific archives for the ESA Astronomy, Planetary and Heliosphysics missions. The SAS is focused on the needs from the scientific community. In this context, the SAS will provide access to the most valuable scientific metadata coming from the EAS-DPS through a set of public releases. According to the policy of public releases defined by the Euclid Consortium, the plan is to deliver 3 data releases to the scientific community every 2 years after the nominal start of the mission [6].  The design of the EAS is fully hardware independent and partially software independent. Metadata storage is based on a RDBMS, currently Oracle 11g, but it will be possible to switch to other RDBMS in the future. Metadata storage in the DPS and the SAS are completely independent and can be based on different solutions. DSS server can use  local or NFS-mounted filesysytem, Grid SE, sftp server, iRODS, Astro-WISE dataserver and can be extended for the existing or upcoming storage cases.  The ability to extend DSS storage practicaly to any storage solution was introduced in the design to cope with the possible changes during the lifetime of the project and to support the SGS for at least 8 years after the launch.  The design of the SAS follows the latest generation of archives being developed by the ESDC, taking full advantage of the existing knowledge, expertise and code. The SAS builds on top of the latest ESDC’s common Archives Building System Infrastructure (ABSI), which defines the common components to the latest ESA Science Archives (i.e. Gaia) within a three-tier modular architecture (client, server and data layer).  Figure 2: SAS Architecture Diagram 4.  EAS INTERFACES There are a number of interfaces and services which are implemented for EAS. EAS DPS will have to support interaction with SDC for the data processing (through Infrastructure Abstract Layer- IAL). IAL queries EAS DPS for commands for the data distribution and data processing and ingest metadata for newly created data products. The basic metadata browsing service allows to browse content of the metadata storage (according to the visibility scope for each user) and retrieve metadata for the data product in XML or other format.  DPS will be supplied with at least 2 other services – for the manual inspection of the quality of data products and for the retrieval and update of calibration frames.   The SAS will provide two ways of access through the Archive User Services (AUS), a web-based portal and a command line interface for programming access. The web-portal is based on the Google Web Toolkit technology [7] and it has been designed to be easy to use and to provide a friendly interaction.   Figure 3: SAS Web User Interface On the server side, the components are based on Java and it  will integrate standard VO protocols to manage requests from the users. At database level, the metadata repository will support the Science Exploitation Data Model (SEDM), that describes the metadata and catalogues oriented to scientific exploration. At the time of writing, the RDBMS is PostgreSQL join with pgSphere and Q3C modules that provides spherical data types, functions and operators to PostgreSQL.  Finally, the Metadata Transfer Service (MTS) will ingest the science metadata of the data release compliant with the SEDM from the ECDM at DPS, according to the policy of public releases defined by the Euclid Consortium.  The scientific requirements on the SAS mainly covers three different areas: parametric search for metadata and catalogues, data retrieval and visualization of images, spectra, etc. Regarding the maps visualization, it will be based on the technology developed for the ESA Sky [8], that allows the exploration of the astronomical resources using a useful and intuitive web interface. In addition, it will provide a set of tools to allow on-line research.  For the data retrieval, SAS stores the links pointing to the data products as part of its metadata. That links are managed by the DSS servers at the SDCs to deliver the data to the end users. VO protocols play an important role within the architecture of the SAS, but also supporting added value tools integrated into the archive. Table Access Protocol (TAP+) [9] that provides efficient parametric search, has been developed for the Gaia Archive (Reference to Gaia BIDS Contribution), is also part of the SAS infrastructure and can be accessed from the Web interface.    Figure 4: TAP Web Interface in SAS VOSpace [10], the IVOA protocol for distributed data storage, is also integrated as part of the SAS infrastructure as an added value tool for the archive. It provides a storage abstraction layer and sharing capabilities transparent for the user.   Figure 5: VOSpace Web User Interface Other VO protocols like SAMP, will provide interoperability with astronomical analysis applications like Topcat among others.  To conclude, SAS will provide the tools and VO interfaces to enable the Software-to-Data Paradigm and "bring the software to the data" for the Euclid science.  4.  EAS STATUS The EAS Prototype was developed and tested in 2013 and 2014,. In 2015 the prototype formed the basis of the first version of EAS itself.  First interfaces for the DPS and the DSS were released and tested during IT challenges organized by Euclid Consortium producing and storing simulated Euclid data. We have successfully tested massive metadata ingestion for the data objects  with extensive data lineage (KIDS DR1) [11]. In 2015 EAS team tested 2-copies configuration of EAS DPS  metadata storage and metadata transfer between EAS DPS  metadata storage mirrors in ESAC and Groningen.  5. FUTURE DEVELOPMENT In the following years prior to the start of the mission the EAS will go through a number of crucial steps in the development including the final selection of an RDBMS. We plan as well to create a system which will support a dynamic data model:  accommodating  changes in ECDM without re-creation of the metadata storage scheme and migrating the data between different versions  of the data model. Current DPS metadata storage will be separated on DPS metadata storage proper and DSS metadata storage. 6. REFERENCES [1] R. Laureijs et al., Euclid Definition Study Report, arxiv1110.3193, 2011 [2] O.R.Williams et al., Data transmission, handling and dissemination of Euclid, Proceedings of NETSPACE 2014, 11, 2014 [3]  F. Pasian et al., Development Plans for the Euclid Science Ground Segment, Proceedings of ADASS XXIII, ASP Conf. Ser. 485, 505, 2014 [4]   K.Begeman et al., The Astro-WISE datacentric information system, Experimental Astronomy 35, 1, 2013   [5]  K.Begeman et al., LOFAR Information System, Future Generation Computer Systems 27, 319, 2011  [6] Maurice P. et al., Euclid : Big Data From Dark Space, Big Data Spain, Oct. 2015 [7] Dewsbury, R., Google Web Toolkit Applications, ISBN 978-0-321-50196-7, Prentice Hall, (2007). [8] Salgado, J., ESA Sky: A Tool to Discover Astronomical Data from ESA Missions, Big Data From Space 2016.  [9] Dowler P. et at., IVOA Recommendation: Table Access Protocol Version 1.0, 2011arXiv1110.0497D [10] Graham M. et al., IVO Working Draft : VOSpace Version 2.1, WD-VOSpace-2.1-20150601 [11] J.T.A. De Jong et al., Kilo Degree Survey, Experimental Astronomy 35, 24, 2013   

University of Groningen Research Database

  
 University of Groningen
The Euclid Archive System: A Datacentric Approach to Big Data
Belikov, Andrey; Williams, Owen; Altieri, Bruno; Boxhoorn, Danny; Buenadicha, Guillermo ;
Droge, Bob; McFarland, John; Nieto, Sara; Salgado, Jesus; de Teodoro, Pilar
Published in:
Proceedings of 2016 conference on Big Data from Space (BiDS'16)
DOI:
10.2788/854791
IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from
it. Please check the document version below.
Document Version
Early version, also known as pre-print
Publication date:
2016
Link to publication in University of Groningen/UMCG research database
Citation for published version (APA):
Belikov, A., Williams, O., Altieri, B., Boxhoorn, D., Buenadicha, G., Droge, B., ... Valentijn, E. (2016). The
Euclid Archive System: A Datacentric Approach to Big Data. In P. Soille, & P. G. Marchetti (Eds.),
Proceedings of 2016 conference on Big Data from Space (BiDS'16) (Vol. JRC100655, pp. 212-215).
Publications Office of the European Union . https://doi.org/10.2788/854791
Copyright
Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the
author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).
Take-down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately
and investigate your claim.
Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the
number of authors shown on this cover page is limited to 10 maximum.
Download date: 21-05-2019
THE EUCLID ARCHIVE SYSTEM:   
A  DATA-CENTRIC APPROACH TO BIG DATA 
A.N.Belikov (1), O.R.Williams(2), B.Altieri(3), D.Boxhoorn(1) , G.Buenadicha(3), B. Droege(2),   
J.McFarland(1), S.Nieto(3), J. Salgado(3), P. de Teodoro(3), A.Tsyganov(2), E.A. Valentijn(1) 
(1) Kapteyn Astonomical Institute, University of Groningen, Groningen, The Netherlands 
(2) Donald Smits Centre for Information Technology, University of Groningen, Groningen, The Netherlands 
(3) European Space Astronomy Center, European Space Agency, Spain 
 
ABSTRACT 
The Euclid Archive System (EAS) is in the core of the 
Euclid Science Ground Segment (SGS). It supports   the 
processing and storage of  Euclid data from the input 
raw frames to the science-ready images and catalogs. We 
review the architectural design of the system, 
implementation progress and main challenges in the 
building of  the EAS.   
Index Terms— Euclid, Data Storage, 
Information System 
1. INTRODUCTION 
Euclid is an ESA M2 mission and a milestone in the 
understanding of the geometry of the Universe [1]. Euclid 
faces two  main challenges from the point of view of the 
data processing.  Firstly, the unprecedented accuracy which 
must be achieved in order to meet the scientific goals. 
Secondly, the mission will depend heavily on the processing 
and reprocessing of the ground-based data which will form 
the bulk of the data volume to store [2]. In total Euclid will 
produce up to 26 PB per year of observations [3]. 
2.  EAS AND SGS 
The Euclid Science Ground Segment is a distributed data 
processing and data storage system which is responsible for 
the delivery of the science-ready data to ESA [3]. The SGS 
is formed by 9 national Science Data Centres and the 
EuclidScience Operations Centre. Each data centre provides 
resources for processing, along with expertize in the coding 
of Euclid pipelines.  The Euclid Archive System (EAS) is a 
core element of the SGS implementing data-centric 
approach [4] to the data processing.  
In the SGS and the EAS the data is a combination of data 
files (images, spectra, catalogs) and metadata. The metadata 
includes the full  data lineage, which is necessary to achive 
the goals of the Euclid mission.   The EAS must provide to 
SGS users and subsystems of SGS the ability to trace any bit 
of information produced in the SGS and must  provide 
quality information on the produced images, spectra and 
catalogs. According to our experience in previous missions 
(Astro-WISE [4] and LOFAR Long-Term Archive [5])  the 
data volume of metadata will not exceed 5% of the total data 
volume. The clear difference between the bulk of the data 
stored in files and extensive metadata describing data 
products in files allows to move most of data mining 
operations to the relational database without access to the 
data files.      
The EAS is a joint development between ESA and Euclid 
Consortium led by the SDC of the Netherlands and the 
ESDC (ESAC Science Data Centre) . 
It should be stressed that the EAS is not an archive in the 
conventional sense. Instead, the EAS is a distributed 
scientific information system which ensures that any 
operation with the data is registered and can be traced back 
to the user and  pipeline.  EAS ensures that SGS can access 
and process hundreds of Petabytes of data.  
3.  EAS DESIGN 
 Figure 1: Overview of EAS and it's place in Euclid SGS 
The EAS design is based on requirements, formed by the 
ESA and the Euclid community, on the data processing in 
distributed environment. To allow the handling of  
numerous and often contradictory requirements on the EAS 
the whole system was divided on 3 independent parts: 
 The Data Processing System (DPS) for the interactions 
within the SGS 
 The Science Archive System (SAS) for the delivery of 
data releases, their long-term preservation and support 
of Euclid scientific use-cases,  
 The distributed Storage System (DSS) for the storage of 
data files. 
Figure 1 shows overview of the EAS design with primary 
users and their interaction with DPS, SAS and DSS.  
The primary task of the DPS is to server as an information 
system for the production of the data release. To achieve 
this goal DPS  keeps data lineage for each data object 
tracing any operation in the system to allow data 
reproduction and reprocessing. DPS allows to trace each 
frame, catalog or source catalog back to the original raw 
data, processing parameters used and pipeline which 
produced it. As well it allows to see the current status of the 
processing in SGS and quality of the raw and processed 
data.   
The DSS serves both the  DPS and SAS as a common 
distributed file storage solution.  In this solution we utilize a 
distributed approach for the data files storage and a 
centralized approach to  the storage of the metadata.  The 
DSS consists of DSS servers which are interfacing the non-
homogeneous data storage solutions provided by the 
national SDCs. At least one DSS server is installed at each 
SDC and stores the data files processed or created during 
running of a pipeline in this SDC. The design of the 
processing plan for each pipeline allows to minimize data 
file transfer between SDCs. To ensure zero loss of the data 
at least 2 copies of each data file is created in DSS. Each 
copy is registered in the metadata storage.  
The EAS implements the Euclid Common Data Model 
(ECDM) which describes both scientific data (data products 
generated by pipelines) and processing and operational 
metadata (processing and data distribution orders, location 
of the file at DSS, processing plan). The ECDM is 
implemented ``as it is'' inside the DPS and transformed to 
Science Exploration Data Model for the implementation in 
SAS.  The interface between SAS and DPS will allow to 
SAS to retrieve the science metadata from the DPS for the 
Euclid data release and put it in long-term storage in SAS. 
The same ECDM is used to bind pipeline data flows.  
The SAS is part of the EAS and it's aimed to support the 
needs for scientific data exploration for Euclid Consortium 
and wider  astronomical community. The SAS, as part of the 
Euclid Archive, will face the challenge of Big Data, as it 
will store a huge and increasing amount of scientific 
metadata, and catalogues of 10 billion of galaxies. This will 
provide the worldwide astronomical community with an 
extremely large source of targets for future missions. Under 
these premises, the SAS will face the challenge to guarantee 
the preservation and public access of the data stored to the 
scientific community. 
 
The SAS is being built at the ESAC Science Data Centre 
(ESDC), that is in charge of the development and operations 
of the scientific archives for the ESA Astronomy, Planetary 
and Heliosphysics missions. The SAS is focused on the 
needs from the scientific community. In this context, the 
SAS will provide access to the most valuable scientific 
metadata coming from the EAS-DPS through a set of public 
releases. According to the policy of public releases defined 
by the Euclid Consortium, the plan is to deliver 3 data 
releases to the scientific community every 2 years after the 
nominal start of the mission [6]. 
 
The design of the EAS is fully hardware independent and 
partially software independent. Metadata storage is based on 
a RDBMS, currently Oracle 11g, but it will be possible to 
switch to other RDBMS in the future. Metadata storage in 
the DPS and the SAS are completely independent and can 
be based on different solutions. DSS server can use  local or 
NFS-mounted filesysytem, Grid SE, sftp server, iRODS, 
Astro-WISE dataserver and can be extended for the existing 
or upcoming storage cases.  The ability to extend DSS 
storage practicaly to any storage solution was introduced in 
the design to cope with the possible changes during the 
lifetime of the project and to support the SGS for at least 8 
years after the launch.  
The design of the SAS follows the latest generation of 
archives being developed by the ESDC, taking full 
advantage of the existing knowledge, expertise and code. 
The SAS builds on top of the latest ESDC’s common 
Archives Building System Infrastructure (ABSI), which 
defines the common components to the latest ESA Science 
Archives (i.e. Gaia) within a three-tier modular architecture 
(client, server and data layer). 
 
Figure 2: SAS Architecture Diagram 
4.  EAS INTERFACES 
There are a number of interfaces and services which are 
implemented for EAS. EAS DPS will have to support 
interaction with SDC for the data processing (through 
Infrastructure Abstract Layer- IAL). IAL queries EAS DPS 
for commands for the data distribution and data processing 
and ingest metadata for newly created data products. The 
basic metadata browsing service allows to browse content of 
the metadata storage (according to the visibility scope for 
each user) and retrieve metadata for the data product in 
XML or other format.  
DPS will be supplied with at least 2 other services – for the 
manual inspection of the quality of data products and for the 
retrieval and update of calibration frames.   
The SAS will provide two ways of access through the 
Archive User Services (AUS), a web-based portal and a 
command line interface for programming access. The web-
portal is based on the Google Web Toolkit technology [7] 
and it has been designed to be easy to use and to provide a 
friendly interaction.  
 
Figure 3: SAS Web User Interface 
On the server side, the components are based on Java and it  
will integrate standard VO protocols to manage requests 
from the users. 
At database level, the metadata repository will support the 
Science Exploitation Data Model (SEDM), that describes 
the metadata and catalogues oriented to scientific 
exploration. At the time of writing, the RDBMS is 
PostgreSQL join with pgSphere and Q3C modules that 
provides spherical data types, functions and operators to 
PostgreSQL. 
 
Finally, the Metadata Transfer Service (MTS) will ingest the 
science metadata of the data release compliant with the 
SEDM from the ECDM at DPS, according to the policy of 
public releases defined by the Euclid Consortium. 
 
The scientific requirements on the SAS mainly covers three 
different areas: parametric search for metadata and 
catalogues, data retrieval and visualization of images, 
spectra, etc. Regarding the maps visualization, it will be 
based on the technology developed for the ESA Sky [8], that 
allows the exploration of the astronomical resources using a 
useful and intuitive web interface. In addition, it will 
provide a set of tools to allow on-line research. 
 
For the data retrieval, SAS stores the links pointing to the 
data products as part of its metadata. That links are managed 
by the DSS servers at the SDCs to deliver the data to the end 
users. 
VO protocols play an important role within the architecture 
of the SAS, but also supporting added value tools integrated 
into the archive. Table Access Protocol (TAP+) [9] that 
provides efficient parametric search, has been developed for 
the Gaia Archive (Reference to Gaia BIDS Contribution), is 
also part of the SAS infrastructure and can be accessed from 
the Web interface.  
  
Figure 4: TAP Web Interface in SAS 
VOSpace [10], the IVOA protocol for distributed data 
storage, is also integrated as part of the SAS infrastructure 
as an added value tool for the archive. It provides a storage 
abstraction layer and sharing capabilities transparent for the 
user. 
 
 
Figure 5: VOSpace Web User Interface 
Other VO protocols like SAMP, will provide 
interoperability with astronomical analysis applications like 
Topcat among others. 
 
To conclude, SAS will provide the tools and VO interfaces 
to enable the Software-to-Data Paradigm and "bring the 
software to the data" for the Euclid science. 
 
4.  EAS STATUS 
The EAS Prototype was developed and tested in 2013 and 
2014,. In 2015 the prototype formed the basis of the first 
version of EAS itself.  First interfaces for the DPS and the 
DSS were released and tested during IT challenges 
organized by Euclid Consortium producing and storing 
simulated Euclid data. We have successfully tested massive 
metadata ingestion for the data objects  with extensive data 
lineage (KIDS DR1) [11]. 
In 2015 EAS team tested 2-copies configuration of EAS 
DPS  metadata storage and metadata transfer between EAS 
DPS  metadata storage mirrors in ESAC and Groningen.  
5. FUTURE DEVELOPMENT 
In the following years prior to the start of the mission the 
EAS will go through a number of crucial steps in the 
development including the final selection of an RDBMS. 
We plan as well to create a system which will support a 
dynamic data model:  accommodating  changes in ECDM 
without re-creation of the metadata storage scheme and 
migrating the data between different versions  of the data 
model. Current DPS metadata storage will be separated on 
DPS metadata storage proper and DSS metadata storage. 
6. REFERENCES 
[1] R. Laureijs et al., Euclid Definition Study Report, 
arxiv1110.3193, 2011 
[2] O.R.Williams et al., Data transmission, handling and 
dissemination of Euclid, Proceedings of NETSPACE 2014, 11, 
2014 
[3]  F. Pasian et al., Development Plans for the Euclid Science 
Ground Segment, Proceedings of ADASS XXIII, ASP Conf. Ser. 
485, 505, 2014 
[4]   K.Begeman et al., The Astro-WISE datacentric information 
system, Experimental Astronomy 35, 1, 2013   
[5]  K.Begeman et al., LOFAR Information System, Future 
Generation Computer Systems 27, 319, 2011 
 [6] Maurice P. et al., Euclid : Big Data From Dark Space, Big 
Data Spain, Oct. 2015 
[7] Dewsbury, R., Google Web Toolkit Applications, ISBN 978-0-
321-50196-7, Prentice Hall, (2007). 
[8] Salgado, J., ESA Sky: A Tool to Discover Astronomical Data 
from ESA Missions, Big Data From Space 2016. 
 [9] Dowler P. et at., IVOA Recommendation: Table Access 
Protocol Version 1.0, 2011arXiv1110.0497D 
[10] Graham M. et al., IVO Working Draft : VOSpace 
Version 2.1, WD-VOSpace-2.1-20150601 
[11] J.T.A. De Jong et al., Kilo Degree Survey, Experimental 
Astronomy 35, 24, 2013  
 


We present the architecture for the planned German Copernicus collaborative data access and exploitation infrastructure. It shall enable the use of operational earth observation data from the Copernicus ground segment for applications and services developed by commercial, public and scientific institutions in Germany. 

Copernicus – the European Earth Observation programme Global Monitoring for Environment and Security – is an ambitious undertaking of the European Commission, the European Space Agency, EUMETSAT and all its member states, to support European citizens, decision makers, scientists and industry with a constant and reliable stream of up-to-date information measured by Earth orbiting satellites. 

Due to the high data volume of Sentinel and other future missions, access to Sentinel data in form of download and data distribution service is not enough. The infrastructure also has to provide hosted processing capacity following the Big-Data paradigm of "transferring the software to the data" that can be cost-effectively used by many projects and services in Germany and beyond

Reck, Christoph

Campuzano, Gina

Dengler, Klaus

Heinen, Torsten

Winkler, Mario

Institute of Transport Research:Publications

GERMAN COPERNICUS DATA ACCESS AND EXPLOITATION INFRASTRUCTURE  Christoph Reck, Gina Campuzano, Klaus Dengler, Torsten Heinen, Mario Winkler  German Aerospace Center (DLR), Earth Observation Center (EOC), Oberpfaffenhofen, Germany   ABSTRACT We present the architecture for the planned German Copernicus collaborative data access and exploitation infrastructure. It shall enable the use of operational earth observation data from the Copernicus ground segment for applications and services developed by commercial, public and scientific institutions in Germany. Copernicus – the European Earth Observation programme Global Monitoring for Environment and Security – is an ambitious undertaking of the European Commission, the European Space Agency, EUMETSAT and all its member states, to support European citizens, decision makers, scientists and industry with a constant and reliable stream of up-to-date information measured by Earth orbiting satellites. Due to the high data volume of Sentinel and other future missions, access to Sentinel data in form of download and data distribution service is not enough. The infrastructure also has to provide hosted processing capacity following the Big-Data paradigm of "transferring the software to the data" that can be cost-effectively used by many projects and services in Germany and beyond.  Index Terms – Earth Observation, Online Data Access, Copernicus, Sentinel mission datasets, Big-Data, CODE-DE 1. INTRODUCTION In 2013, the Space Administration of the German Aerospace Center (DLR) asked the DLR Earth Observation Center (EOC) to conceive an architectural concept for a   Figure 1: Context of the German national Copernicus data access and exploitation collaborative infrastructure national platform to mirror incoming Sentinel data for use and offering computing services to its users. The primary goal of the Copernicus collaborative data access and exploitation infrastructure is to provide a platform to exploit the possibilities of the continuous data stream of “free, full and open” Copernicus Sentinel data with the following elements: • Ingestion and Archive • Search and Access • Portal and User Management • Processing and Value Added Products • Monitoring and Reporting 2. CHALLENGE Data rates of all incoming Sentinel-1, Sentinel-2, Sentinel-3 and Sentinel-5p earth observation user products itself pose the biggest challenge:   2014 2015 2016 2017 2018 2019 2020 Yearly volume [TB] 180 966 4 490 6 591 7 250 7 469 8 127 Average Data Rate [Mbit/s] 194 257 1 194 1 753 1 928 1 987 2 162 Figure 2: Estimated data volume and rates for ingesting all Sentinel user products However, the outbound rate is far larger when this data is systematically processed with different algorithms and accessed by several end users. This requires extremely performant infrastructure and data access methods. 3. ARCHITECTURE The proposed architecture is highly modular being composed of individual services, which form a general-purpose system, largely based on existing software components, put together in an innovative manner to serve as a high-performant and scalable platform. Figure 3 below depicts the complete architectural concept that is briefly described in the next paragraphs. Partners & Cloud ProvidersCopernicus data Access and exploitation Collaborative InfrastructureSearch & PortrayalPortalWebsiteServiceMarketplaceServiceProvisioningArchive ProcessingExternal EntitiesApplications& ProjectsCollaborativeAcquisition StationNRT Data ProvisioningCatalogueServiceNationalMirrorHostedProcessingApplicationsServicesHostingNational ArchiveLTAPast DataAccessCopernicus Space Component – Data AccessCore PDGS Data Hub CataloguesThird-Party DataMonitoringReportingNetworkUser Mgmt.GovernanceLocalStorageData SourcesAccessDistribution ServiceDownloadServiceSubscriptionServiceIngestionServiceIaaS(Hosted Proc.)Applications  Users Figure 3: Architectural Concept Search and Portrayal Portal Website provides the access point for the collaborative infrastructure elements, portrayed from the registered services, such as the catalogue of the archived products, application provisioning monitoring, etc. A Service Marketplace allows the collaborative infrastructure, external users and projects to register and advertise their accessible services (e.g. a third-party catalogue or an algorithm). The Service Provisioning enables the users to deploy and launch their applications. Archive National Mirror provides fast access to data products for global, regional or time-series processing and re-processing. The products provided over the national mirror are the same as disseminated by ESA to the member states. Local Storage archives the value-added and convenience products of the collaborative infrastructure. Catalogue Service for the inventory and metadata of the Copernicus Core Services and convenience products available in the Archive. It shall permit dataset discovery visualization and selection, with pointers to the download location. Access Ingestion Service is in charge of populating the National Mirror from the Data Hub. Subscription Service manages registered filters on incoming datasets to provide triggering capabilities for data-driven applications. A trigger may fire an email, dissemination to the users, or drive a data driven workflow. Distribution Service is in charge of dissemination with equal priority the Sentinel data to the application modules (push service). This contrasts to the capability for direct access to the National Mirror or Local Storage which is subject to polling and workflow delays (pull service). Download Service provides the means for the users to access products stored in the online archive. The download will provide direct HTTP access to the data products [OGC 13-043] or tailoring and subsetting capabilities via the OGC WCS standard. Processing Applications are deployed and run within the collaborative infrastructure. Among them are services and processors for the systematic and on-demand generation of convenience products from Sentinel data. Services Hosting provides the collaborative infrastructure or deploying and running system applications (e.g. all of the above listed services) and hosted applications for the users. In principle this is a cloud environment with virtual machines and suitable software tools and web interface. Hosted Processing provides a platform for massive-parallel earth observation data processing, for deploying processors in a cost- and resource-effective way on a cluster close to the data, and job handling with the data selection and processor execution. Also monitoring and accounting functionality is attributed to this element. Supporting Services Other cross-cutting services like Governance, User Management, Monitoring and Reporting, Backup and Network infrastructure complement the architecture to its completeness. 4. SOFTWARE SYSTEMS DLR EOC designed a system utilizing known best breed software and hardware systems assembled to a streamlined simple, scalable and performant architecture covering all interfaces from Discovery over Visualization to Download for users with novel clients: • Fast catalogue with HMA CSW and OGC OpenSearch interfaces • Flexible dataset browsing with OGC Web Map Service (WMS) • High performance data access using HTTP protocol • Advanced data access using OGC Web Coverage Service (WCS) • Parallel file system on an online storage attached network (SAN) • Redundant hardware The system shall additionally enable retrieval of historical data from the archive. Discovery The EOWEB® catalogue is based on a database, a metadata-model, ingestion and operation interfaces and user service interfaces compiled for performance [1]. It is configured to hold OGC EOP metadata [4] and provides an HMA CSW standard compliant interface [5] that allows novel clients and user interfaces to comfortably search for data (EOWEB GeoPortal, FEDEO, mapshup, EOxClient). Visualization Geospatial Data Access Service (GDAS) provides the tools and components to register, describe, access, search and retrieve geospatial data. It is manly composed of: • GeoServer (http://geoserver.org) is an open source server for sharing, processing and editing geospatial data. It offers all major OGC standards services needed for this project (WMS, WFS, WCS, CSW), and also can be setup for INSPIRE conformance. • GeoWebCache (http://geowebcache.org/) is used to cache geospatial data (e.g. for WMS) and therefore speed up the access of this data for the clients. • PostgreSQL (http://www.postgresql.org) is an object-relational database with the PostGIS extension it handles spatial-referenced data. • Nginx (http://nginx.org/) is a web server with a strong focus on high concurrency, performance and low memory usage. It is also used for the proxy and download server. Catalog Client The envisaged EOX catalog client provides a as “zoom-in on the data” solution to discover, view, and download available EO data. A timeline and advanced search capabilities allow additional filtering, e.g. by sun angle or cloud coverage.  Figure 4: Catalog Client with result details and download links Download The large storage allows on-line access to the data products.  We have benchmarked the Nginx server and the underlying filesystem to prove the capability of serving files at a total rate of more than 2 GBytes/sec using parallel transfers. Nginx was chosen for its performance as well as its simple configuration model that provides existing extension modules as load-balancing, access control and on-the-fly unzipping and content retrieval. Historical data from the long term archive will be made equally accessible such that these appear to be nearly on-line. Distribution To minimize the load accessing recently ingested data, we are designing an efficient data distribution service based on UFTP (http://uftp-multicast.sourceforge.net/) for an multicast file transfer mechanism. Each distributed file is preceded by a metadata record that allows the client side filter element to decide whether a data product is needed. At the end of the reception, the client library launches a command that has been configured for the filter condition. Processing Bringing the processing to the data is enabled by directly interfacing a processing cluster to the storage area. Processing control and the user interface is allocated in the virtualized or cloud environments. Projects and users can process on the platform interacting on different levels ranging from • Use of data processors and generic processing workflows and toolbox operators available via the Web GUI, producing products from Copernicus data. • Simple data access from their VMs, interactive work with tools available (e.g. Sentinel Toolbox) or provided by the project. • Submission of processing jobs to the cluster, control of processing workflows (orchestration) with platform tools (scripting) or project-specific workflow engines; use of infrastructure services (e.g. subscription) for systematic data-driven processing of new input data; small-scale processing on the VM or (bulk) processing on the cluster. • Deployment of own processors in Docker containers, or as a service in a VM. • Provision of data or processing service to other users with publishing in the Service Marketplace. The processing environment is based on Apache Hadoop using the Calvalus environment designed by Brockman Consult GmbH, enhanced with Docker (deployment), Apache Mesos (resource manager), Marathon (service control and scheduling) and Chronos (batch processing). User Management A central use database and authentication system avoids redundancy and provides single sign on (SSO) capability to the users for accessing infrastructure services after a single login. The user management service consists of the following subsystems: • LDAP Administration Service, which allows an operator to manage user accounts. • The User Registration Service for new users. • The User Self-Management Service allowing a user to manage his own data. • The CAS Server providing interfaces for authentication of users and services, as well as issuing and validating tickets for the SSO process. 4. HARDWARE INFRASTRUCTURE The infrastructure hardware configuration consists of a virtualized environment, a high performance computing cluster, and the storage server:                      2 x 12 Cores E5-2680v3128 D. wamGPFS ServerVMClient(s)VM2 x 12 Cores E5-2680v3128 D. wamGPFS ServerVMClient(s)VM2 x 12 Cores E5-2680v3128 D. wamGPFS ServerVMClient(s)VMStorage Extension 1up to 16 x 8 Dbit/secCC connectionsup to 2 x 16 x 6 Dbit/secSAS connectionsStorage Extension 2Storage Extension 3up to 336 3.5in Disks > 1 t.ytex2 x2x2 x2x2 x2...HtC Cluster Storage Head – Dual Controller2 x Xeon 6 Core 2.6 DHz 48 D. wam2 x Xeon 6 Core 2.6 DHz 48 D. wam2 x Xeon 6 Core 2.6 DHz 48 D. wam... Figure 5: High-Performance server environment • 3 servers with 2 Intel Xeon E5-2680v3 12 Core 2.5GHz, 128 GB RAM, 2 * 300 GB 10k SAS Disks, 2 FC HBA 8 Gbit/s, Quad Port 10 GE NIC, Quad Port 1 GE NIC, 2 hot swappable 750 watt power supplies. • 7 HPC servers with 2 Intel Xeon 6 Core 2.6 GHz, 48 GB RAM, 500 GB SATA Disk, Dual Port 10 GE NIC, Dual Port 1 GE NIC, 2 hot swappable power supplies. • Storage using a GPFS file system with 835 TB in RAID 6 over 282 4 TB NL-SAS disks, dual controllers in one head. Expandable to 12 Petabyte with 4 heads. • Public Cloud within the DSI vCloud environment. Network and interfacing components are not depicted. The hosts are connected via 10 Gbit/s network segments. The external connection starts with a dedicated 3.5 Gbit/s connection interfaced to the Internet and GEANT networks by the DFN X-Win provider. 5. CONCLUSION The German Copernicus Data Access and Exploitation Infrastructure is based on the architecture, software elements and hardware infrastructure outlined in this paper. Most of the system components are re-using existing open-source software, components developed for the DLR multi-mission German Remote Sensing Data Center and the evolution complemented by the OPUS-GMES project that prototyped the: • utilization of near real time and offline (catalogued) earth observation data • development and implementation of end to end value adding services • establishment of state of the art product and service-access • demonstration of representative production ready service chains The hardware infrastructure has been setup and tested within the ESA Data Hub Relay project to ensure its capacity and suitability for the high throughput of the envisaged system. Benchmarks showed the capability of serving data to more than 50 clients with a total data rate beyond 2 GBytes/second in the initial minimal setup. Therefore the presented infrastructure elements have been tested and the risk has been minimized in preparation for the upcoming official German CODE-DE (Copernicus Data and Exploitation Platform – Deutschland) project. Remaining challenges in setting up the operational infrastructure are harmonizing the means for configuration handling of the individual elements, interfacing with the SSO, and the additional effort on the extensions for quota management, prioritization, and ensuring that the security requirements are fulfilled. SELECTED REFERENCES [1] P. Harms, S. Kiemle, D. Dietrich, Extensible Earth Observation Data Catalogues with multiple Interfaces, PV2007 [2] K. Dengler, T. Heinen, A. Huber, K. Molch, E. Mikusch, The EOC Geoservice: Standardized Access to Earth Observation Data Sets and Value Added Products , PV2013 [3] C. Reck, K. Molch, E. Mikusch, J. Hoffmann, Copernicus Data Access and Exploitation Collaborative Infrastructure – COPACI System Concept, DLR 2014 [4] OGC Earth Observation Metadata profile of Observations & Measurements (EOP O&M), OGC 10-157r3, Version 1.0.0, 2012-06-12  [5] European Space Agency (ESA), (2012) Heterogeneous Missions Accessibility (HMA), ISBN  978-92-9221-883-6, http://esamultimedia.esa.int/multimedia/publications/TM-21/TM-21.pdf 

German Copernicus Data Access and Exploitation Platform

Proceedings - University of Groningen

   University of GroningenThe Euclid Archive System: A Datacentric Approach to Big DataBelikov, Andrey; Williams, Owen; Altieri, Bruno; Boxhoorn, Danny; Buenadicha, Guillermo ;Droge, Bob; McFarland, John; Nieto, Sara; Salgado, Jesus; de Teodoro, PilarPublished in:Proceedings of 2016 conference on Big Data from Space (BiDS'16)DOI:10.2788/854791IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionEarly version, also known as pre-printPublication date:2016Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Belikov, A., Williams, O., Altieri, B., Boxhoorn, D., Buenadicha, G., Droge, B., McFarland, J., Nieto, S.,Salgado, J., de Teodoro, P., Tsyganov, A., & Valentijn, E. (2016). The Euclid Archive System: ADatacentric Approach to Big Data. In P. Soille, & P. G. Marchetti (Eds.), Proceedings of 2016 conferenceon Big Data from Space (BiDS'16) (Vol. JRC100655, pp. 212-215). Publications Office of the EuropeanUnion . https://doi.org/10.2788/854791CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 10-09-2023THE EUCLID ARCHIVE SYSTEM:   A  DATA-CENTRIC APPROACH TO BIG DATA A.N.Belikov (1), O.R.Williams(2), B.Altieri(3), D.Boxhoorn(1) , G.Buenadicha(3), B. Droege(2),   J.McFarland(1), S.Nieto(3), J. Salgado(3), P. de Teodoro(3), A.Tsyganov(2), E.A. Valentijn(1) (1) Kapteyn Astonomical Institute, University of Groningen, Groningen, The Netherlands (2) Donald Smits Centre for Information Technology, University of Groningen, Groningen, The Netherlands (3) European Space Astronomy Center, European Space Agency, Spain  ABSTRACT The Euclid Archive System (EAS) is in the core of the Euclid Science Ground Segment (SGS). It supports   the processing and storage of  Euclid data from the input raw frames to the science-ready images and catalogs. We review the architectural design of the system, implementation progress and main challenges in the building of  the EAS.   Index Terms— Euclid, Data Storage, Information System 1. INTRODUCTION Euclid is an ESA M2 mission and a milestone in the understanding of the geometry of the Universe [1]. Euclid faces two  main challenges from the point of view of the data processing.  Firstly, the unprecedented accuracy which must be achieved in order to meet the scientific goals. Secondly, the mission will depend heavily on the processing and reprocessing of the ground-based data which will form the bulk of the data volume to store [2]. In total Euclid will produce up to 26 PB per year of observations [3]. 2.  EAS AND SGS The Euclid Science Ground Segment is a distributed data processing and data storage system which is responsible for the delivery of the science-ready data to ESA [3]. The SGS is formed by 9 national Science Data Centres and the EuclidScience Operations Centre. Each data centre provides resources for processing, along with expertize in the coding of Euclid pipelines.  The Euclid Archive System (EAS) is a core element of the SGS implementing data-centric approach [4] to the data processing.  In the SGS and the EAS the data is a combination of data files (images, spectra, catalogs) and metadata. The metadata includes the full  data lineage, which is necessary to achive the goals of the Euclid mission.   The EAS must provide to SGS users and subsystems of SGS the ability to trace any bit of information produced in the SGS and must  provide quality information on the produced images, spectra and catalogs. According to our experience in previous missions (Astro-WISE [4] and LOFAR Long-Term Archive [5])  the data volume of metadata will not exceed 5% of the total data volume. The clear difference between the bulk of the data stored in files and extensive metadata describing data products in files allows to move most of data mining operations to the relational database without access to the data files.      The EAS is a joint development between ESA and Euclid Consortium led by the SDC of the Netherlands and the ESDC (ESAC Science Data Centre) . It should be stressed that the EAS is not an archive in the conventional sense. Instead, the EAS is a distributed scientific information system which ensures that any operation with the data is registered and can be traced back to the user and  pipeline.  EAS ensures that SGS can access and process hundreds of Petabytes of data.  3.  EAS DESIGN  Figure 1: Overview of EAS and it's place in Euclid SGS The EAS design is based on requirements, formed by the ESA and the Euclid community, on the data processing in distributed environment. To allow the handling of  numerous and often contradictory requirements on the EAS the whole system was divided on 3 independent parts:  The Data Processing System (DPS) for the interactions within the SGS  The Science Archive System (SAS) for the delivery of data releases, their long-term preservation and support of Euclid scientific use-cases,   The distributed Storage System (DSS) for the storage of data files. Figure 1 shows overview of the EAS design with primary users and their interaction with DPS, SAS and DSS.  The primary task of the DPS is to server as an information system for the production of the data release. To achieve this goal DPS  keeps data lineage for each data object tracing any operation in the system to allow data reproduction and reprocessing. DPS allows to trace each frame, catalog or source catalog back to the original raw data, processing parameters used and pipeline which produced it. As well it allows to see the current status of the processing in SGS and quality of the raw and processed data.   The DSS serves both the  DPS and SAS as a common distributed file storage solution.  In this solution we utilize a distributed approach for the data files storage and a centralized approach to  the storage of the metadata.  The DSS consists of DSS servers which are interfacing the non-homogeneous data storage solutions provided by the national SDCs. At least one DSS server is installed at each SDC and stores the data files processed or created during running of a pipeline in this SDC. The design of the processing plan for each pipeline allows to minimize data file transfer between SDCs. To ensure zero loss of the data at least 2 copies of each data file is created in DSS. Each copy is registered in the metadata storage.  The EAS implements the Euclid Common Data Model (ECDM) which describes both scientific data (data products generated by pipelines) and processing and operational metadata (processing and data distribution orders, location of the file at DSS, processing plan). The ECDM is implemented ``as it is'' inside the DPS and transformed to Science Exploration Data Model for the implementation in SAS.  The interface between SAS and DPS will allow to SAS to retrieve the science metadata from the DPS for the Euclid data release and put it in long-term storage in SAS. The same ECDM is used to bind pipeline data flows.  The SAS is part of the EAS and it's aimed to support the needs for scientific data exploration for Euclid Consortium and wider  astronomical community. The SAS, as part of the Euclid Archive, will face the challenge of Big Data, as it will store a huge and increasing amount of scientific metadata, and catalogues of 10 billion of galaxies. This will provide the worldwide astronomical community with an extremely large source of targets for future missions. Under these premises, the SAS will face the challenge to guarantee the preservation and public access of the data stored to the scientific community.  The SAS is being built at the ESAC Science Data Centre (ESDC), that is in charge of the development and operations of the scientific archives for the ESA Astronomy, Planetary and Heliosphysics missions. The SAS is focused on the needs from the scientific community. In this context, the SAS will provide access to the most valuable scientific metadata coming from the EAS-DPS through a set of public releases. According to the policy of public releases defined by the Euclid Consortium, the plan is to deliver 3 data releases to the scientific community every 2 years after the nominal start of the mission [6].  The design of the EAS is fully hardware independent and partially software independent. Metadata storage is based on a RDBMS, currently Oracle 11g, but it will be possible to switch to other RDBMS in the future. Metadata storage in the DPS and the SAS are completely independent and can be based on different solutions. DSS server can use  local or NFS-mounted filesysytem, Grid SE, sftp server, iRODS, Astro-WISE dataserver and can be extended for the existing or upcoming storage cases.  The ability to extend DSS storage practicaly to any storage solution was introduced in the design to cope with the possible changes during the lifetime of the project and to support the SGS for at least 8 years after the launch.  The design of the SAS follows the latest generation of archives being developed by the ESDC, taking full advantage of the existing knowledge, expertise and code. The SAS builds on top of the latest ESDC’s common Archives Building System Infrastructure (ABSI), which defines the common components to the latest ESA Science Archives (i.e. Gaia) within a three-tier modular architecture (client, server and data layer).  Figure 2: SAS Architecture Diagram 4.  EAS INTERFACES There are a number of interfaces and services which are implemented for EAS. EAS DPS will have to support interaction with SDC for the data processing (through Infrastructure Abstract Layer- IAL). IAL queries EAS DPS for commands for the data distribution and data processing and ingest metadata for newly created data products. The basic metadata browsing service allows to browse content of the metadata storage (according to the visibility scope for each user) and retrieve metadata for the data product in XML or other format.  DPS will be supplied with at least 2 other services – for the manual inspection of the quality of data products and for the retrieval and update of calibration frames.   The SAS will provide two ways of access through the Archive User Services (AUS), a web-based portal and a command line interface for programming access. The web-portal is based on the Google Web Toolkit technology [7] and it has been designed to be easy to use and to provide a friendly interaction.   Figure 3: SAS Web User Interface On the server side, the components are based on Java and it  will integrate standard VO protocols to manage requests from the users. At database level, the metadata repository will support the Science Exploitation Data Model (SEDM), that describes the metadata and catalogues oriented to scientific exploration. At the time of writing, the RDBMS is PostgreSQL join with pgSphere and Q3C modules that provides spherical data types, functions and operators to PostgreSQL.  Finally, the Metadata Transfer Service (MTS) will ingest the science metadata of the data release compliant with the SEDM from the ECDM at DPS, according to the policy of public releases defined by the Euclid Consortium.  The scientific requirements on the SAS mainly covers three different areas: parametric search for metadata and catalogues, data retrieval and visualization of images, spectra, etc. Regarding the maps visualization, it will be based on the technology developed for the ESA Sky [8], that allows the exploration of the astronomical resources using a useful and intuitive web interface. In addition, it will provide a set of tools to allow on-line research.  For the data retrieval, SAS stores the links pointing to the data products as part of its metadata. That links are managed by the DSS servers at the SDCs to deliver the data to the end users. VO protocols play an important role within the architecture of the SAS, but also supporting added value tools integrated into the archive. Table Access Protocol (TAP+) [9] that provides efficient parametric search, has been developed for the Gaia Archive (Reference to Gaia BIDS Contribution), is also part of the SAS infrastructure and can be accessed from the Web interface.    Figure 4: TAP Web Interface in SAS VOSpace [10], the IVOA protocol for distributed data storage, is also integrated as part of the SAS infrastructure as an added value tool for the archive. It provides a storage abstraction layer and sharing capabilities transparent for the user.   Figure 5: VOSpace Web User Interface Other VO protocols like SAMP, will provide interoperability with astronomical analysis applications like Topcat among others.  To conclude, SAS will provide the tools and VO interfaces to enable the Software-to-Data Paradigm and "bring the software to the data" for the Euclid science.  4.  EAS STATUS The EAS Prototype was developed and tested in 2013 and 2014,. In 2015 the prototype formed the basis of the first version of EAS itself.  First interfaces for the DPS and the DSS were released and tested during IT challenges organized by Euclid Consortium producing and storing simulated Euclid data. We have successfully tested massive metadata ingestion for the data objects  with extensive data lineage (KIDS DR1) [11]. In 2015 EAS team tested 2-copies configuration of EAS DPS  metadata storage and metadata transfer between EAS DPS  metadata storage mirrors in ESAC and Groningen.  5. FUTURE DEVELOPMENT In the following years prior to the start of the mission the EAS will go through a number of crucial steps in the development including the final selection of an RDBMS. We plan as well to create a system which will support a dynamic data model:  accommodating  changes in ECDM without re-creation of the metadata storage scheme and migrating the data between different versions  of the data model. Current DPS metadata storage will be separated on DPS metadata storage proper and DSS metadata storage. 6. REFERENCES [1] R. Laureijs et al., Euclid Definition Study Report, arxiv1110.3193, 2011 [2] O.R.Williams et al., Data transmission, handling and dissemination of Euclid, Proceedings of NETSPACE 2014, 11, 2014 [3]  F. Pasian et al., Development Plans for the Euclid Science Ground Segment, Proceedings of ADASS XXIII, ASP Conf. Ser. 485, 505, 2014 [4]   K.Begeman et al., The Astro-WISE datacentric information system, Experimental Astronomy 35, 1, 2013   [5]  K.Begeman et al., LOFAR Information System, Future Generation Computer Systems 27, 319, 2011  [6] Maurice P. et al., Euclid : Big Data From Dark Space, Big Data Spain, Oct. 2015 [7] Dewsbury, R., Google Web Toolkit Applications, ISBN 978-0-321-50196-7, Prentice Hall, (2007). [8] Salgado, J., ESA Sky: A Tool to Discover Astronomical Data from ESA Missions, Big Data From Space 2016.  [9] Dowler P. et at., IVOA Recommendation: Table Access Protocol Version 1.0, 2011arXiv1110.0497D [10] Graham M. et al., IVO Working Draft : VOSpace Version 2.1, WD-VOSpace-2.1-20150601 [11] J.T.A. De Jong et al., Kilo Degree Survey, Experimental Astronomy 35, 24, 2013   

1). History of the Virtual Astronomical Observatory. Retrieved from VAO: http://virtualobservatory.org/whatis/history.aspx

14). The World’s Technological Capacity to Store, Communicate, and Compute Information. Retrieved from Science Magazine:

17). Astronomy Overload. Retrieved from Space.com:

21). XDF: The Extensible Data Format Based on XML Concepts. Retrieved from NASA:

XML-Based Approaches for the Integration of Heterogeneous Bio-Molecular Data. In