Challenges of Harmonizing 40 Years of AVHRR Data: The TIMELINE Experience

Earth Observation satellite data allows for the monitoring of the surface of our planet at predefined intervals covering large areas. However, there is only one medium resolution sensor family in orbit that enables an observation time span of 40 and more years at a daily repeat interval. This is the AVHRR sensor family. If we want to investigate the long-term impacts of climate change on our environment, we can only do so based on data that remains available for several decades. If we then want to investigate processes with respect to climate change, we need very high temporal resolution enabling the generation of long-term time series and the derivation of related statistical parameters such as mean, variability, anomalies, and trends. The challenges to generating a well calibrated and harmonized 40-year-long time series based on AVHRR sensor data flown on 14 different platforms are enormous. However, only extremely thorough pre-processing and harmonization ensures that trends found in the data are real trends and not sensor-related (or other) artefacts. The generation of European-wide time series as a basis for the derivation of a multitude of parameters is therefore an extremely challenging task, the details of which are presented in this paper

Data Handling and Preservation for the TanDEM-X Satellite Mission

The main purpose of TanDEM-X – a German SAR (Synthetic Aperture Radar) interferometric satellite mission – is to produce a world-wide consistent and reliable digital elevation model (DEM) with accuracy to the HRTI-3 (High-Resolution Terrain Information) specifications. To reach this goal, the existing TerraSAR-X Mission will be extended by a second radar satellite, which is almost similar to the already flying one. A world-wide network of receiving stations is necessary to downlink the acquired SAR-data with a volume of 350 TB in two and a half year. In addition, also the Payload Ground Segment must be updated to manage the new workflows and the high amount of data up to 2000 TB to produce the global DEM with a volume of approximately 15 TB

NOAA AVHRR Data Curation and Reprocessing - TIMELINE

2013 marks the 35th anniversary of NOAA's Advanced Very High Resolution Radiometer (AVHRR) first launched in 1978. The four to six band multi-spectral AVHRR data constitute a valuable data source for deriving time series of surface parameters, such as snow cover, land surface temperature, or vegetation indices for monitoring global change. AVHRR data are residing in various archives worldwide. ESA has archived NOAA data in several of their facilities; DLR has been receiving and processing NOAA AVHRR data into value-added products since 1981. In order to properly preserve this valuable dataset, ESA has initiated the NOAA AVHRR data curation and reprocessing initiative as a pilot project within ESA's long-term data preservation (LTDP) program. The project's objective is the consolidation and curation of a temporally and geographically complete and consistent dataset of 1 km NOAA AVHRR HRPT data over Europe and Canada. The project will be conducted jointly with the members of the LTDP working group of the ESA Ground Segment Coordination Body. Additionally, in 2013 DLR has kicked off TIMELINE, a project focusing on generating a range of over 25 thematic products from the complete coverage of 30 years worth of NOAA AVHRR 1 km level 0 data over Europe. The thematic products will serve as input for performing higher level time series analyses on land surface dynamics, such as investigating changes in annual snow cover duration or exploring local land surface temperature trends. Within the project a sustainable, generic infrastructure for processing Earth observation time series at DLR will be implemented. With their wide scope, the two projects touch upon a variety of topics and disciplines on the scientific side as well as in large volume data processing and data management. The initial challenge lies in identifying and closing temporal and geographic data gaps for consolidating and curating 30 years worth of AVHRR data held in dispersed archives. Curation will be done in line with the European LTDP data preservation workflow. Preparation for systematic pre-processing and processing will involve external calibration and navigation data to be introduced in the pre-processing systems. Turning scientific algorithms into flexible processing systems to efficiently re-process 30 terabytes of level 0 data into about 50 terabytes of validated thematic products is another challenge. On the data management side product archiving and access infrastructures will have to be adapted to ensure e.g. efficient and user-friendly retrieval of localized time series data stacks. Interactive on-the-fly visualizations of time series products will facilitate the understanding of complex and interacting temporal phenomena. Close collaboration between scientists and ground segment engineers - and their respective operational approaches - is a particularly useful aspect of this project, with a view to expanding ground segment services towards incorporating flexible, large volume scientific processing chains in the future. From the NOAA AVHRR Data Curation and Reprocessing and TIMELINE projects novel ideas for managing large Earth observation time series data sets are expected across the entire end-to-end chain - from data consolidation via reprocessing and data management to innovative ways of data discovery and exploitation

Operating Tool for a Distributed Data and Information Management System

The German Remote Sensing Data Center has developed the Data Information and Management System - DIMS [1], a multi-mission ground system that went into operation early 2000 with the Shuttle Radar Topography Mission SRTM and supports multiple upcoming programs and missions. DIMS provides ground system services for earth observation product processing, archiving, ordering and delivery. DIMS successfully uses newest technologies within its services. During the next years more than 50 processing systems for earth observation data are to be integrated and set up for operations in DIMS. Efficient and homogenous administration and operation of complex information systems is a fundamental aspect in modern system design. Experiences from earlier archiving system developments allowed us to develop a set of scalable services and frameworks that can be centrally administered. Reuse of software components saves costs in current and future missions and increases uniformity during the operational phase. This paper presents the DIMS Operating Tool as the solution taken to simplify operation tasks for this large and distributed system

Management of "Future Products" in the Distributed Payload Ground Segment for TerraSAR-X

The German Remote Sensing Data Center (DLR-DFD) has developed the Data and Information Management System (DIMS) as a distributed multi-mission infrastructure which will be reused within the TerraSAR-X Payload Ground Segment. Besides other requirements like large data volume, time and reliability constraints a major issue is the management of "future products" throughout the system. For TerraSAR-X DIMS has to support the ordering of user specified future acquisitions, to handle the interface to Mission Planning, to control the subsequent reception of corresponding raw data and to manage the processing and delivery of the derived products. The requested features will be incorporated into the existing modular components in a way that they are reusable also for further missions. The paper describes the concept of DIMS how to integrate the satellite in the loop and how to meet the additional requirements in a distributed system

Dataflow and Workflow Organization - The Data Management for the TerraSAR-X Payload Ground Segment

The payload ground segment (PGS) for the recently launched German radar satellite, TerraSAR-X, performs the operational data management of the acquired satellite data. This comprises well-known functions such as reception, systematic and ondemand processing, archiving and cataloguing, ordering, and dissemination of digital Earth-observation products. In addition, it comprises new functions like large-scale multimode acquisition ordering by users, integration with a commercial service segment, and new interfaces and workflows within the complete ground segment. The TerraSAR-X PGS is based on the Data Information and Management System (DIMS), the multimission data-handling infrastructure of the German Remote Sensing Data Center (DFD) at the German Aerospace Center. The development and integration of the new functions and complex workflows for TerraSAR-X were achieved and successfully tested on time. After the support of commissioning phase for five months, the system is now operational. As an intended side effect, the PGS for TerraSAR-X is, in several aspects, a pattern being reused for upcoming future missions, thus substantially reducing overall developmental costs. This paper investigates features of the TerraSAR-X PGS that enable the reuse in a multimission environment. It summarizes the achieved enhancements and extensions of DIMS to support the TerraSAR-X mission. Special emphasis is placed on the implementation of the request workflow initiated by user orders and the corresponding data flow within the distributed DFD multimission facility

Data Information and Management System for Earth Observation Products based on CORBA and Java

The Data Information and Management System (DIMS) is a new generation facility from DLR-DFD to handle earth observation products. DIMS is a multi-mission ground system and will be configured to support the tasks performed within the framework of the upcoming programs/missions such as the Shuttle Radar Topography Mission (SRTM), EUMETSAT Polar System (EPS) or ENVISAT. The DIMS components follow a modular design based on CORBA AND Java, providing: o Communication independant from platform and programming language o Relocatable and distributed services o Reusable basic components used to build up application components o Platform independent software DIMS is responsible for user information services (including on-line delivery), off-line delivery, post-processing, the product library, ordering control and production control. A cornerstone is the product library which is the source of processing input and the destination of the processing output. It provides a complete and consistent reference to the configured products. Based on the product library, ordering control manages the business process ordering throughout the system. A unified operating tool provides interactive access for the above components. Interfaces to related systems like EPS ground segment, Unified Meteorological Archive and Retrieval Facility (UMARF) or ENVISAT Payload Data Segment (PDS) are foreseen to exchange products, catalogue and order information. The near real-time, as well as the off-line services, are provided by DIMS. The experience gained at DFD in managing large amounts of meteorological and earth observation data and a modern system design, using state-of-the-art technologies, resulted in the valuable features offered by DIMS: o Multi-mission facility, supporting products from many sensors/satellites/missions in parallel o User information services including guide, directory, inventory, browse and ordering o Product library, including automatic robot archive and inventory with optimized spatial access methods o Robot archive, with an extendable capacity of 300Tbyte o Interfacing of local and remote processors o Product delivery on media or via Internet o Support of distributed sites based on interoperability with CORBA The software is foreseen for installation and configuration at other institutions

A Description of the Data-Driven SAR Data Workflow in the TerraSAR-X Payload Ground Segment

TerraSAR-X is a national German satellite providing a high-resolution X-Band SAR instrumentation featuring StripMap, ScanSAR and SpotLight imaging modes in various polarizations. The experimental full polarization mode is carried out using a dual receive antenna capability. An active lifetime of five years is anticipated, the launch is expected in 2006. The TerraSAR-X mission shall serve both commercial and scientific needs. It is implemented in the frame of a public-private partnership between EADS Astrium GmbH (providing the TerraSAR-X Space Segment) and the German Aerospace Center DLR (providing the TerraSAR-X Ground Segment). The commercial product exploitation rights are with Infoterra GmbH, the scientific ones remain at DLR. The Infoterra business concept foresees the inclusion of a Direct Access Service through which TerraSAR-X data may be directly received by stations either for the sole purpose to produce geo-information products for their own and private applications or to generate and distribute commercial products. Essential tasks of the TerraSAR-X Payload Ground Segment developed (and later operated) by DLR are the SAR payload data reception, their archiving and processing, and the distribution of the generated SAR products to users. This paper summarizes the TerraSAR-X Payload Ground Segment design. It describes the SAR data workflow ranging from the payload data reception at the Neustrelitz Ground Station to the long-term data archiving in the multi-mission Product Library at the German Remote Sensing Data Center. Supplementary data as provided by the TerraSAR-X Mission Operation Segment and used during the data-driven generation of the SAR level 0 products are described together with their dissemination concept. The adaptability of the chosen approach for the operation of a Direct Access Station is addressed