Many NASA Earth Observing System (EOS) have either already reached the end of their active life or are nearing it. Preservmissionsation of data products is a fairly well defined task for the NASA EOS Data Centers or DAACs.The Goddard Earth Sciences Data and Information Services Center (GES-DISC) has implemented a repository system, which is capable of long-term archive of documentation artifacts and other associated digital content. The existing GES-DISC Repository System is based on Fedora Commons, an open-source repository management software, for cost savings and flexibility.The first mission to utilize the GES-DISC Repository System was the High Resolution Dynamics Limb Sounder (HIRDLS) on the Aura spacecraft. Since then, the GES DISC has gathered documentation from the UARS and TOMS into the Repository. The Microwave Limb Sounder (MLS) team has begun delivering some early pre-launch documents to the GES-DISC Repository System as well. Other missions in planning or progress include AIRS, OMI, SORCE, SNPP Sounder, and TRMM

Deshong, Barbara E.

Gerasimov, Irina V.

Johnson, J.

Khayat, Mohammad G.

Wei, J.

Wysk, Joseph D.

NASA Technical Reports Server

Data Preservation
Final Step in the Life Cycle of a Mission
J. Johnson1,2, J. Wei1 , M. Khayat1,2, J. Wysk1,2, B. Deshong1,2, I. Gerasimov1,2
1 NASA Goddard Space Flight Center    2 ADNET Systems Inc.
NASA/Goddard Earth Sciences Data and Information Services Center (GES DISC)
https://disc.gsfc.nasa.gov
Many NASA Earth Observing System (EOS) have either already reached the end of
their active life or are nearing it. Preservmissionsation of data products is a fairly
well defined task for the NASA EOS Data Centers or DAACs. However,
supporting documentation and other artifacts from these missions are also
critical to the long-term studies of our planet’s climate, and to aid future
generation’s ability to understand climatic changes. The challenge is how to
preserve these items along with the traditional data products.
The Goddard Earth Sciences Data and Information Services Center (GES-DISC)
has implemented a repository system, which is capable of long-term archive of
documentation artifacts and other associated digital content. The existing GES-
DISC Repository System is based on Fedora Commons, an open-source repository
management software, for cost savings and flexibility.
The first mission to utilize the GES-DISC Repository System was the High
Resolution Dynamics Limb Sounder (HIRDLS) on the Aura spacecraft. Since
then, the GES DISC has gathered documentation from the UARS and TOMS into
the Repository. The Microwave Limb Sounder (MLS) team has begun delivering
some early pre-launch documents to the GES-DISC Repository System as well.
Other missions in planning or progress include AIRS, OMI, SORCE, SNPP
Sounder, and TRMM.
The GES-DISC uses Fedora Commons, an open-source repository management
system that is used in many universities, research centers, and libraries. It comes with
a simple web-based GUI interface which provides for easy administration of the
system. The GUI also allows one to enter objects or datastreams (these can be of any
type document, image, source code, binary data, etc.) into the system. The system
uses XML to manage the objects. The GES-DISC has also developed a command
line script to allow batch ingest of objects into the Fedora Repository.
External users access the publicly available documents by visiting the mission
specific documentation page for that instrument. The Fedora repository system is at
the backend and makes access to the linked documents possible. Note that restricted
objects (ITAR, proprietary, or software) are not accessible through the public
interface. Four missions are now public:
MLS https://disc.gsfc.nasa.gov/information/documents?title=MLS%20Mission%20Preservation%20Documents
HIRDLS https://disc.gsfc.nasa.gov/information/documents?title=HIRDLS%20Mission%20Preservation%20Documents  
TOMS https://disc.gsfc.nasa.gov/information/documents?title=TOMS%20Mission%20Preservation%20Documents 
UARS https://disc.gsfc.nasa.gov/information/documents?title=UARS%20Mission%20Preservation%20Documents
Being able to understand and interpret data from these older missions after
the experts familiar with them have moved on or are no longer available is a
concern to NASA. The importance of preserving the Earth Science documentation
and data resulted in the issuance of the “NASA Earth Science Data Preservation
Content Specification” (423-SPEC-001) for the Earth Science Data and Information
System (ESDIS) supported data centers. Documents and data in the GES-DISC
repository are archived and classified according to 423-SPEC-001 into 9 categories
listed in the table below for each mission.
Available at https://earthdata.nasa.gov/files/NASA_ESD_Preservation_Spec.pdf
Introduction
NASA Earth Science Data Preservation Content Specification
432-SPEC-001
Fedora Commons Interface
Public Access of Preservation Documents
GES DISC Data Preservation Implementation
Overview of the physical
objects sorting, tagging, storage in
archive and distribution system. 
Restricted documents are currently only
available by contacting the GES DISC
user services.
1) Identify documentation
GES DISC Science Support staff identified specific information needed
permission in    Data Preservation Mission List by working closely with the
original mission teams to sort out documents for preservation
2) Specify and implement preservation environment
a) Local archive based on open-source system Fedora Commons
b) Implementation is complete for HIRDLS, TOMS and UARS missions
c) Other missions MLS, OMI, AIRS, SORCE, TRMM, etc. are in progress
3) Retrieve documentation
Public documents accessed by users from GES DISC web site
4) Implement retrieval and distribution services
a) Access for internal GES-DISC users
b) External Access via WWW for unrestricted (public) documents
c) External Access for restricted documents (ITAR) via User Services contact
d) Iterate with other DAACs/community
Lessons Learned, Challenges, Future
 Heritage missions require extensive work to identify and classify documents
 Restricted (ITAR or Proprietary) requires special  handling vs. Unrestricted
 Incorporate DOI metadata into repository (if available)
 Replace or upgrade Fedora Commons which is currently 5 years old
 Move Repository to a NASA ESDIS-wide Repository? 
Internal GUI for ingest and archive records into
the repository (above), and internal GUI to search
and retrieve items from the repository (right).  
Restricted
Documents
Locked 
Offline
Help 
Desk
Aura STM 2019
I. Category 
Preflight/Pre-
Opcrations Cal ibration 
Science Data Products 
Science Data Product 
Documentation 
Mission Data 
Calibration 
Sciem;e Data Product 
Software 
Science Data Product 
Algor ithm Inp uts 
Science Data Product 
Validation 
Science Dala Software 
Tools 
2. Content Item 
lnslrument Description 
Preflight/Pre-operationa l 
Calibration Data 
Raw Data aod Deri ved 
Products 
Metadata 
Product T earn 
Product Requirements 
Processing and Algori thm 
Version History 
Product Generation 
Algorithm 
Product Qua li ty 
Product Applicat ion 
Cal ibration Method 
Calibration Data 
Science data product 
generation software a11d 
soft\vare documentat ion 
Anci llary dnta and 
documentation 
Datasets and documentation 
Software and documentation 
J. Definition/Description 
Documentation of Instrument/sensor characteristics inc luding pre-flight or pre-operational pcrfom,ancc measurements (e.g., spectra l response, 
instrument geometric calibration (geo- location offsets), noise characteristi cs, etc .). 
Numeric (digital data) fil es of Instrument/sensor characteristics including pre-flight or pre-operational perfonnance measurements (e.g., spectra l 
response, instrument geometric calibration (gco-location offsets), no iBc characteristics, etc.). 
Raw data are data values at full resolution as direct ly measured hy a spacehome, ai rhome or in situ instrument. Derived products are hi gher level 
products (level 1 b through 4) where calibration and gco-locnt.ion tra nsfom1ations have been applied to generate sensor units, nndlor algorithms 
have been applied to generate gridded geophysical parameters. 
lnfonnation about data to facilitate discovery, search, access, understanding and usage associated with each of the data products. 
Names of key science team leads and product team members (deve lopment, belp desk and operations), roles, performing organization, contact 
information, sponsoring agencies or organizations and comments about the products. 
Requ irements and designs fo r each science data product e ither explicitly or by reference to the requi rements/design documents. Product 
requirements and designs should include contem, formal , latency, accuracy and qual ity. 
For al l products he ld in the archive, documentation of process ing history and production version history, indicating wh ich versi ons were used 
\Vhen, why different versions came about, and what the improvements were from version to version . For all products held in the archive, the 
versions of source code used to produce the products should be avai lable at the archive. 
Detai led di sc ussion of processing algorithms, outputs, error budgets and limitations. Processing algorithms and their theoretical (scien tific and 
mathematical) basis, including complete description of any sampling or mapping algorithm used in creation of the prod uct, geo-location, 
radiometric ca libration, geophysica l parameters, samp ling or mapping algorithms used in creation of the product, algorithm software 
documentation, & bigh-kvcl data flov,· diagrams. Description of how tbc algorithm is numerically implemented. 
Description of the impact to product quality due to issues w ith computationally intensive operations (e.g., large matrix inversions, truncation and 
rounding). Documentation of product quality assessment (methods used, assessment summaries for each vers ion of the datasets). Description of 
embedded data at the granule level including quality flags, product data uncertainty fie lds, data issues logs, etc. Relevant test reports, reviews, and 
appraisals. 
Useful references to publi shed articles about the use of the data and user feedbac k received by the science and instrument teams about the products. 
Includes rcpo11s of any peculiarities or notab le features observed in the products. 
T bc method s u~cd for instrument/sensor radiometric and geometric calibration while in operation (e.g., in orb it) . The source code used in applying 
the calibration algorithms. Documentation of in-line changes to calibration or to instrument or platform operations or condi tions that occur 
throughout the mission. 
Instrument and platform engineering data collected during operations (e.g., on orbit), including platform and instrumenl environment, events and 
maneuvers; attitude and ephemeris; a ircraft position; acquis ition logs that record data gaps; calibration look-up tables; ai1d any significant externaJ 
event data that may have impacted the observations. 
Source code used to generate products al all levels in the science data process ing system. Software release notes, including references to versions 
of operating systems, compilers, commercial software libraries used in the code. Versions of science duta product software should be arch ived for 
each major product release. A major product release is characterized by the appearance of peer reviewed pub lications where reported results arc 
based on the product version. 
Complete info rmation on any anc illary data or other data sets used in generation or ca lib rati on of the data set or derived product, either expl icitly in 
data descriptions or by reference to approp riate publications. Anc ill ary data shoLtld be stored with the products un less it is available from another 
pennanent archive facility. 
Accuracy of products, as measured by va li dation testing, and compared to accuracy requirements . Description of validation process, including 
id~ntification of val idation data sets, measurement protocols, data collection, analysis and accuracy reporting . 
Product access (reader) too ls. Software source code that would fac ilitate use of the calibration data, ancillary data and the data products at a ll 
levels. Inclu des sothvare source code useful for creating programs that will read and display the calibration data , ancillary data and product data 
and metadata values. Commerc ial tools should be identified with appropriate references . 
+ I 81 gs 61 02 dscvm 02/feOO ra;a dmi fl/ lfa pp~ :llf6& 189 't-sel eaed lro:J ex ~ 1 & ~b54·se lectedlnde x~ 3 .., 2! ] I t!j-., fedorHutori al 
Firefox -., [:) http://g,; ... /Mmlnl* X [J Search Aep(',s itory );( "· . User Malling Ust .. X :X, Example SOAP 0 1... X .X REST API Tests · F .. X :::J ~le:,1/,t ... Test l.xml X ' i' . 
Open Obj ect 
"' 
O~tastream 10 I sorn.oot, rrod.J :t:Sotm.,.:,oc 
Labe l ,w,a,a .,ooa,.Jr 
Co:r:rcl Croup [ -..9.d Cont•"' (HJ T] h portObjed 
MME Type , pp1,.,;,,,v,Jr ~ v; • • ubj.,.1 , .,, 
Create Object 
c,-nt. NuObJ-
lng~• I Obi@d 
Fcrrrat FO"""-·l ,l 
I lng•st ob~ 
t"• td< ~"'-' H,ctmy tl"""""'"" lool< t!"P 
Stote [ A<tl,o lAl 
Versionab le [ Updateswmaeateanewve,.ion 
Formc t URI 
Altemate lD; IM ;gµqy:p:p 
ch~cksum [ omut ' • ] 
Checksum [__ ::...:.. 
0 c,,-, , nhomaU RL 
Q c,,-,,,.. ~om or,1, 
I Add Dalasl"am 
GI A<; U/AM, 1 ""' lllS C" !!l :~na v,, {!l 4 0, :1•1 o I X ''"' " ' " I ""' '"" ''" "'" 
Pu,oeObjed 
" 
--
· --
Fedora Commons· 
• • 
•\ 
Fields to display: 
' 
pid creator 
labcl fillh.i«-t 
s.tate de.s.crip.tion 
Oillledd publisher 
cllate contributor 
tnOate date 
dcmllitte !J'p_e 
' 
title fuanaJ 
• ru 
. ,;. "ARTt-DATA ,eO , u.• ~ . 0 
GES DISC 11 A Fee·'" ' He, • -
~~ , " " ~ Cl lfil ,,,- ..; 
0 HI Rnl S M issio n P res e, rv,.tion n oc111 n e,nts 
HIRDLS Mission Preservation Documents 
In -' " u ff o, I Lo, ,rnK" m ,K; 1, d ;, :,n ·a " L,Kl ,k " """" " '" ';as rnb.,a j lu If,-, lu"""' ~w.o.'H R [;l.$ n ,; , ,, ;"" w e n m mnk;sg ,.,,.;lnb' " a, i;c,;,,, u f a,II 
l'" U ;<-1, o<.cas •• ;UI• d,~." o, Lo ., ;1 , U ~ L I , Sc lu Ll , a a , , u., I Ju ,.a, '" , ,, •. T l, • du ,arn o , lo l,a•• [.,,,, ""!•' "' "" ., cc u;d ;, ~ Lu Lf w ~A3A E .,,J , 
C,c lonc• Cata F,oca.-.aj oa Con»nt c-,..cnoat "" I< SS· S 'E'O-Uu · ,. F lasca oaf, , to th is d,,,,m·an foe ch• "'"" I p,eca.-.aj on gold., , ,., 
and poHo;o, of cho C:JD IG ,upp,,,,,d nb , ;,,, I yea hove • prnb on· ..,,,; ., ng acy of lh, d o,umoota Hao.cl See,, , loo,o ",-d a, o -naH to 
our lelp Desk i~slc-l" el:-.:: 'sci!>l:m .oo,a ,Qov\ w 'lh ch , name ef <re c·orn-ie,1 a,d ao, lnformo1on deser blio 1Se o·ol: err ;-ou 
'"'""' '' ' ''"" ' 
"I. P reflight Calibration 
1.1 Instrument D@scription 
R~,.,;, • ., a: Lo 
• ' ·' ""'~"' 1,eoul,e~.,cc, u ,,,,,,.,,, ' " ,u:: SG ·C.11 : .o ]SL po· 
• P co jh •"" ''' ' am I ;,.,, ,C .•,at" v ,., ;fb' lrn A-a,,l , e· a a""" TC HIR ' ' " F•II 
•~POI S A <l"' ''"" ~,, .,;11rnc: TC r :P 1',7a F" 
• S .. ,a l Prooc~"' " vr Rc Ulc• o, c " I' D-'" " LI "' HI P.DLS ., ,,,u,·,e 11 L. le ,·cu '1 8 . .,JI 
• SC,D co lnD F1c .. ,aow, M"rt,: T O·NGA· O"-""' 
• """ , .,,,,, ~ ,,,,,, ,ad L~S Ui,ee:lc a : t,: .o,· .o,Ob.pdt 
• r"'l=c••• c,,· ,m "" " F •m, "' '°"~ 1n ,oc,.1 •.c., , TC -OXF I IO '"" 
• ., am o e a rt., eCa~e o f ~1-.r, S ah oe,,, ' roo, ~ e,,., · T~ CX F " ""' 
, ,, ' 0'-.1o poi 
~s Ou It 
• ,;,,, s , os·,~:a ~ ,c;~-,,;; , u ue _uss.m 
• l o. r l,:i, t Cs ll, ,a· c , SuC,y <tsn I r e:, CI OP _ 1· c · g , 
• S pooe V ,o·N A,ert~o {S V.',), CIO P _ SV.',.<ga 
- I mLO Lh, ed " · l<om, Log, CA ·LGG -O OC rec 
• I mLO r,n , " '"' CL· •n -10T. 4 , 
• A ,-ca;;, Go~; '"'"';'" c ;,., ,~tion 1 r "' •LGC -10e·, bl, D 
Fedora Repository 
Find Objects 
identifier Search all fields for phrase help 
SO.UM 
language 
relation Or search specific field(s): help_ 
Cill'.f!llge 
rights Maximum Results 20 
Search 
SC - HIR- 018L 
HIRD LS 
HIGH R ESOL UT ION DYNAMICS LIM.B SOUN DER 
O riginalor: John C . G ille Date: 19-97- 10-20 
Subj«! / Titk : lnstrumcm Requirements Document (I RD) 
Contents I Dcscr ip1 io n / Summary: 
This documl'm dl'scribcs 1hc TC<(lli rcmr:ms th.al the l-! igh Resolu tion Dynnmics L imb Sounde r 
(H I R.DLS) instrument mus! meet in ordcr 10 smisfy the sc ient ific goals o t" tbc H IR J)LS f' rogrnm. 
Kcy Words: instrument r-cquin:mcnts. inve!ai_gMio11 rcq1 1i m ncnts. IRD 
Purpose (20 character~ ma.11: imuml : specify rcquircmcnl~ 
A pproved / Rc viewrn.l By: 
Date (yyyy- mm- dd): 
Oxford Un i, ·e rsity 
Dep tt rtmcnl of Almosphcric. Oceanic, and 
r1a1w1ary rhy~ ic,; 
Parks Road 
Ox ford OX I J PU, Uni tcd Kingdom 
Univers ity ofColoradc at Boulder 
Center fo r L imb A lmo,;plucri~ Sournl ing 
3300 Mitchell lam.•, Su ite 250 
Boulder, Colorado 80301 - 2296 
United States of Amcricrr 
EOS 
. ' 
Missions 
Documents 
PDF, 
DOC, 
PPT, GIF, 
ASCII, 
data, etc. 
GES-DISC 
Temp 
Stora~c 
Sort & Tag 
Classify Type, Identify Restriction.}.\ 1'1\< 
Propl'ietary), Ingest in Repository 
Electronic 
Repository 
Unrestricted 
Documents 
WWW 
Restricted 
Documents 
https://ntrs.nasa.gov/search.jsp?R=20190030310 2019-09-26T19:53:48+00:00Z


Data Preservation: Final Step in the Life Cycle of a Mission

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server