Evolution of Archival Storage (from Tape to Memory)

Over the last three decades, there has been a significant evolution in storage technologies supporting archival of remote sensing data. This section provides a brief survey of how these technologies have evolved. Three main technologies are considered - tape, hard disk and solid state disk. Their historical evolution is traced, summarizing how reductions in cost have helped being able to store larger volumes of data on faster media. The cost per GB of media is only one of the considerations in determining the best approach to archival storage. Active archives generally require faster response to user requests for data than permanent archives. The archive costs have to consider facilities and other capital costs, operations costs, software licenses, utilities costs, etc. For meeting requirements in any organization, typically a mix of technologies is needed

NASA's Earth Observing System Data and Information System - EOSDIS

This slide presentation reviews the work of NASA's Earth Observing System Data and Information System (EOSDIS), a petabyte-scale archive of environmental data that supports global climate change research. The Earth Science Data Systems provide end-to-end capabilities to deliver data and information products to users in support of understanding the Earth system. The presentation contains photographs from space of recent events, (i.e., the effects of the tsunami in Japan, and the wildfires in Australia.) It also includes details of the Data Centers that provide the data to EOSDIS and Science Investigator-led Processing Systems. Information about the Land, Atmosphere Near-real-time Capability for EOS (LANCE) and some of the uses that the system has made possible are reviewed. Also included is information about how to access the data, and evolutionary plans for the future of the system

Developing a Standard for Earth Observation Data Preservation Content - A Path to Future Usability

For datasets to be usable, many pieces of information in addition to the data themselves are essential. During the active parts of the lifecycle of dataset generating projects, the needed information is usually accessible through individuals familiar with the various aspects of the projects. However, the utility of datasets tends to outlive the lives of projects, by several decades in many cases. Thus it is essential to capture all the relevant information about the datasets, data, metadata and associate knowledge that is sufficient to read, understand, interpret and reuse the datasets, while the projects are still active. The capture and preservation should be such that the data are usable when no consultation is available from the original project participants. Identification of specific categories of content through an international standard is beneficial to the user communities of the future, so that projects involving Earth observations and generating data products can consistently plan for preservation and future usability of the project outcomes. While there are existing standards that address archival and preservation in general, there are no existing international standards or specifications today to address what content should be preserved. The standard, ISO 19165-1, titled "Geographic Information - Preservation of digital data and metadata Part 1: Fundamentals" considers geographic information preservation in general. It acknowledges that "specific content items needed to preserve the full provenance and context of the data and associated metadata depend on the needs of the designated community and types of datasets (e.g., maps, remotely sensed data from satellites and airborne instruments, physical samples). Follow-up parts to this standard may be developed detailing content items appropriate to individual disciplines." NASA proposed an extension to this standard, titled "Geographic information -- Preservation of digital data and metadata -- Part 2: Content specifications for Earth observation data and derived digital products." The development of this extension is in progress with participation by an international team representing nine countries. The purpose of this paper is to introduce this standard and report on its status

Collaborations and Partnerships in NASAs Earth Science Data Systems

NASA has been collecting Earth observation data from spaceborne instruments since 1960. Today, there are tens of satellites orbiting the Earth and collecting frequent global observations for the benefit of mankind. Collaboration between NASA and organizations in the US and other countries has been extremely important in maintaining the Earth observation capabilities as well as collecting, organizing and managing the data. These collaborations have occurred in the form of: 1. NASAs developing and launching spacecraft and instruments for operation by other agencies; 2. Instruments from collaborating organizations being flown on NASA satellites; and 3. Instruments from NASA being flown on satellites from collaborating organizations. In addition, there are collaborations such as joint science teams, data exchanges, and participation in international organizations to promote interoperability of various data systems. The purpose of this paper is to describe some of the Earth science data-related collaborative efforts in which NASA participates, and highlight a few results relevant to Earth system science research obtained through such collaborations

Approach to Managing MeaSURES Data at the GSFC Earth Science Data and Information Services Center (GES DISC)

A major need stated by the NASA Earth science research strategy is to develop long-term, consistent, and calibrated data and products that are valid across multiple missions and satellite sensors. (NASA Solicitation for Making Earth System data records for Use in Research Environments (MEaSUREs) 2006-2010) Selected projects create long term records of a given parameter, called Earth Science Data Records (ESDRs), based on mature algorithms that bring together continuous multi-sensor data. ESDRs, associated algorithms, vetted by the appropriate community, are archived at a NASA affiliated data center for archive, stewardship, and distribution. See http://measures-projects.gsfc.nasa.gov/ for more details. This presentation describes the NASA GSFC Earth Science Data and Information Services Center (GES DISC) approach to managing the MEaSUREs ESDR datasets assigned to GES DISC. (Energy/water cycle related and atmospheric composition ESDRs) GES DISC will utilize its experience to integrate existing and proven reusable data management components to accommodate the new ESDRs. Components include a data archive system (S4PA), a data discovery and access system (Mirador), and various web services for data access. In addition, if determined to be useful to the user community, the Giovanni data exploration tool will be made available to ESDRs. The GES DISC data integration methodology to be used for the MEaSUREs datasets is presented. The goals of this presentation are to share an approach to ESDR integration, and initiate discussions amongst the data centers, data managers and data providers for the purpose of gaining efficiencies in data management for MEaSUREs projects

Sharing, and reusing quality information of individual digital datasets

Open-source science builds on open and free resources that include data, metadata, software, and workflows. Informed decisions on whether and how to (re)use digital datasets are dependent on an understanding about the quality of the underpinning data and relevant information. However, quality information, being difficult to curate and often context specific, is currently not readily available for sharing within and across disciplines. To help address this challenge and promote the creation and (re)use of freely and openly shared information about the quality of individual datasets, members of several groups around the world have undertaken an effort to develop international community guidelines with practical recommendations for the Earth science community, collaborating with international domain experts. The guidelines were inspired by the guiding principles of being findable, accessible, interoperable, and reusable (FAIR). Use of the FAIR dataset quality information guidelines is intended to help stakeholders, such as scientific data centers, digital data repositories, and producers, publishers, stewards and managers of data, to: i) capture, describe, and represent quality information of their datasets in a manner that is consistent with the FAIR Guiding Principles; ii) allow for the maximum discovery, trust, sharing, and reuse of their datasets; and iii) enable international access to and integration of dataset quality information. This article describes the processes that developed the guidelines that are aligned with the FAIR principles, presents a generic quality assessment workflow, describes the guidelines for preparing and disseminating dataset quality information, and outlines a path forward to improve their disciplinary diversity.The development and baseline of the community FAIR-DQI guidelines document would not have been possible without the voluntary and dedicated effort of the domain experts of the International FAIR-DQI Community Guidelines Working Group. We would like to thank all members of the working group for their interest, participation, and contribution.Peer Reviewed"Article signat per 11 autors/es: Ge Peng , Carlo Lacagnina, Robert R. Downs, Anette Ganske, Hampapuram K. Ramapriyan, Ivana Ivánová, Lesley Wyborn, Dave Jones, Lucy Bastin, Chung-lin Shie, David F. Moroni"Postprint (published version

Evolution of the Earth Observing System (EOS) Data and Information System (EOSDIS)

One of the strategic goals of the U.S. National Aeronautics and Space Administration (NASA) is to "Develop a balanced overall program of science, exploration, and aeronautics consistent with the redirection of the human spaceflight program to focus on exploration". An important sub-goal of this goal is to "Study Earth from space to advance scientific understanding and meet societal needs." NASA meets this subgoal in partnership with other U.S. agencies and international organizations through its Earth science program. A major component of NASA s Earth science program is the Earth Observing System (EOS). The EOS program was started in 1990 with the primary purpose of modeling global climate change. This program consists of a set of space-borne instruments, science teams, and a data system. The instruments are designed to obtain highly accurate, frequent and global measurements of geophysical properties of land, oceans and atmosphere. The science teams are responsible for designing the instruments as well as scientific algorithms to derive information from the instrument measurements. The data system, called the EOS Data and Information System (EOSDIS), produces data products using those algorithms as well as archives and distributes such products. The first of the EOS instruments were launched in November 1997 on the Japanese satellite called the Tropical Rainfall Measuring Mission (TRMM) and the last, on the U.S. satellite Aura, were launched in July 2004. The instrument science teams have been active since the inception of the program in 1990 and have participation from Brazil, Canada, France, Japan, Netherlands, United Kingdom and U.S. The development of EOSDIS was initiated in 1990, and this data system has been serving the user community since 1994. The purpose of this chapter is to discuss the history and evolution of EOSDIS since its beginnings to the present and indicate how it continues to evolve into the future. this chapter is organized as follows. Sect. 7.2 provides a discussion of EOSDIS, its elements and their functions. Sect. 7.3 provides details regarding the move towards more distributed systems for supporting both the core and community needs to be served by NASA Earth science data systems. Sect. 7.4 discusses the use of standards and interfaces and their importance in EOSDIS. Sect. 7.5 provides details about the EOSDIS Evolution Study. Sect. 7.6 presents the implementation of the EOSDIS Evolution plan. Sect. 7.7 briefly outlines the progress that the implementation has made towards the 2015 Vision, followed by a summary in Sect. 7.8

Global Community Guidelines for Documenting, Sharing, and Reusing Quality Information of Individual Digital Datasets

Open-source science builds on open and free resources that include data, metadata, software, and workflows. Informed decisions on whether and how to (re)use digital datasets are dependent on an understanding about the quality of the underpinning data and relevant information. However, quality information, being difficult to curate and often context specific, is currently not readily available for sharing within and across disciplines. To help address this challenge and promote the creation and (re) use of freely and openly shared information about the quality of individual datasets, members of several groups around the world have undertaken an effort to develop international community guidelines with practical recommendations for the Earth science community, collaborating with international domain experts. The guidelines were inspired by the guiding principles of being findable, accessible, interoperable, and reusable (FAIR). Use of the FAIR dataset quality information guidelines is intended to help stakeholders, such as scientific data centers, digital data repositories, and producers, publishers, stewards and managers of data, to: i) capture, describe, and represent quality information of their datasets in a manner that is consistent with the FAIR Guiding Principles; ii) allow for the maximum discovery, trust, sharing, and reuse of their datasets; and iii) enable international access to and integration of dataset quality information. This article describes the processes that developed the guidelines that are aligned with the FAIR principles, presents a generic quality assessment workflow, describes the guidelines for preparing and disseminating dataset quality information, and outlines a path forward to improve their disciplinary diversity

Global Community Guidelines for Documenting, Sharing, and Reusing Quality Information of Individual Digital Datasets

Open-source science builds on open and free resources that include data, metadata, software, and workflows. Informed decisions on whether and how to (re)use digital datasets are dependent on an understanding about the quality of the underpinning data and relevant information. However, quality information, being difficult to curate and often context specific, is currently not readily available for sharing within and across disciplines. To help address this challenge and promote the creation and (re)use of freely and openly shared information about the quality of individual datasets, members of several groups around the world have undertaken an effort to develop international community guidelines with practical recommendations for the Earth science community, collaborating with international domain experts. The guidelines were inspired by the guiding principles of being findable, accessible, interoperable, and reusable (FAIR). Use of the FAIR dataset quality information guidelines is intended to help stakeholders, such as scientific data centers, digital data repositories, and producers, publishers, stewards and managers of data, to: i) capture, describe, and represent quality information of their datasets in a manner that is consistent with the FAIR Guiding Principles; ii) allow for the maximum discovery, trust, sharing, and reuse of their datasets; and iii) enable international access to and integration of dataset quality information. This article describes the processes that developed the guidelines that are aligned with the FAIR principles, presents a generic quality assessment workflow, describes the guidelines for preparing and disseminating dataset quality information, and outlines a path forward to improve their disciplinary diversity

NASA EOSDIS Data Identifiers: Approach and System

NASA’s Earth Science Data and Information System (ESDIS) Project began investigating the use of Digital Object Identifiers (DOIs) in 2010 with the goal of assigning DOIs to various data products. These Earth science research data products produced using Earth observations and models are archived and distributed by twelve Distributed Active Archive Centers (DAACs) located across the United States. Each data center serves a different Earth science discipline user community and, accordingly, has a unique approach and process for generating and archiving a variety of data products. These varied approaches present a challenge for developing a DOI solution. To address this challenge, the ESDIS Project has developed processes, guidelines, and several models for creating and assigning DOIs. Initially the DOI assignment and registration process was started as a prototype but now it is fully operational. In February 2012, the ESDIS Project started using the California Digital Library (CDL) EZID for registering DOIs. The DOI assignments were initially labor-intensive. The system is now automated, and the assignments are progressing rapidly. As of February 28, 2017, over 50% of the data products at the DAACs had been assigned DOIs. Citations using the DOIs increased from about 100 to over 370 between 2015 and 2016