The Crustal Dynamics Data Information System (CDDIS) supports data archiving and distribution activities for the space geodesy and geodynamics community. The main objectives of the system are to store space geodesy and geodynamics related data and products in a central archive, to maintain information about the archival of these data,to disseminate these data and information in a timely manner to a global scientific research community, and provide user based tools for the exploration and use of the archive. The CDDIS data system and its archive is a key component in several of the geometric services within the International Association of Geodesy (IAG) and its observing systemthe Global Geodetic Observing System (GGOS), including the IGS, the International DORIS Service (IDS), the International Laser Ranging Service (ILRS), the International VLBI Service for Geodesy and Astrometry (IVS), and the International Earth Rotation and Reference Systems Service (IERS). The CDDIS provides on-line access to over 17 Tbytes of dataand derived products in support of the IAG services and GGOS. The systems archive continues to grow and improve as new activities are supported and enhancements are implemented. Recently, the CDDIS has established a real-time streaming capability for GNSS data and products. Furthermore, enhancements to metadata describing the contents ofthe archive have been developed to facilitate data discovery. This poster will provide a review of the improvements in the system infrastructure that CDDIS has made over the past year for the geodetic community and describe future plans for the system

Michael, Patrick

Noll, Carey

NASA Technical Reports Server

Metadata
IAG 
Service
Archive
Metadata Metadata
IAG 
Service
Archive
IAG 
Service
Archive
(CDDIS)
GGOS Portal
UsersData Services ...
CEN 12/01/2016
GNSS Antenna and 
Satellite
SLR System and 
Target Satellite
VLBI Antenna and 
Celestial Map
DORIS Beacon and 
Satellite Receiver
Space Geodesy Techniques
G41B-1013
Crustal Dynamics Data Information System
Background:
• The Crustal Dynamics Data Information System (CDDIS) is NASA’s active archive of space geodesy data, products, and 
information (Global Navigation Satellite System/GNSS, Satellite Laser Ranging/SLR, Very Long Baseline 
Interferometry/VLBI, and Doppler Orbitography and Radio-positioning Integrated by Satellite/DORIS).
• The CDDIS is funded by NASA/ESDIS but cooperates extensively with the international community.
• The largest CDDIS user community comes from the services within the International Association of Geodesy (IAG). 
• The contents of the CDDIS archive are utilized for geodetic studies, e.g., plate tectonics, earthquake displacements, Earth 
orientation, Earth’s surface deformation, Earth’s gravity field, etc. 
• The CDDIS archive also plays an interdisciplinary role in supporting the derivation of a Terrestrial Reference Frame (the 
foundation for virtually all airborne, space-based and ground-based Earth observations), precise orbit determination (POD) 
for NASA/international missions, atmospheric studies, etc.
Archive Contents:
• Data:
 + Stations in the GNSS, SLR/LLR, VLBI, and DORIS networks generate point data on a multi-day, daily, hourly, and/or 
sub-hourly basis
 + GNSS: 585+ sites tracking GPS, GLONASS, and new GNSS (Galileo, QZSS, Beidou, IRNSS)
 + Laser Ranging (SLR and LLR): ~40 sites tracking 90+ satellites (including the Moon)
 + VLBI: 45 sites
 + DORIS: 58 sites tracking 6 satellites
• Products:
 + Precise network station positions (for ITRF)
 + Satellite orbits (for POD)
 + Station and satellite clocks (for timing)
 + Earth rotation parameters
 + Positions of celestial objects (for CRF)
 + Atmospheric parameters (ionosphere TEC, 
  troposphere ZPD) …
• Metadata information:
 + Non-standard metadata, data type specific
 + Extracted from incoming files
 + Internal access to metadata database
Archive Statistics:
• File size is typically <2Mb/data “granule”, 
 <10Mb/derived product “granule”
• Archive size: ~17.5TB/190M files
• Ingest rate: ~9GB (100K files)/day
• Distribution rate: ~500GB (~4M files)/day 
• Data (L1, L1B), products (L2) derived from 
 these data, and information about data and 
 products
• Multi-day, daily, hourly, sub-hourly
• Varying latencies (minutes, hours, days)
Archive Usage: 
• The CDDIS contains data and derived products from over 1500 observing sites located at about 1000 locations around the 
world, going back in time as far as 1975.
• The archive is updated with new data/product files on varying time scales, dependent on the data type, from a sub-daily 
basis to weekly basis.
• Users require continuous access to data for generation of products on pre-determined schedules.
• The average user of the CDDIS accesses the contents of the archive through anonymous ftp by means of automated scripts 
executed on predefined schedules (typically sub-daily). 
• Analysts can use this method for data transfer because they are familiar with the structure of the online archive and thus 
know what files they require, their availability schedule, and where to find them within the online structure
The CDDIS and the IAG
• CDDIS is the principle data center for the geometric supporting services created under the umbrella of the International 
Association of Geodesy (IAG):
 + International GNSS Service (IGS)
 + International Laser Ranging Service (ILRS)
 + International VLBI Service for Geodesy and Astrometry (IVS)
 + International DORIS Service (IDS)
• These services function as cooperating federations dedicated to a particular type of data (e.g., GNSS, SLR, VLBI, or DORIS)
• The services provide data and products on an operational basis to geodesy analysts as well as a broader scientific 
community and are examples of a successful model of community management
• Successful operation through cooperation of many international organizations who leverage their respective limited 
resources to all levels of service functionality
• The CDDIS user community primarily consists of analysts supporting the services within the International Association of 
Geodesy.
• These groups produce derived products (e.g., positions of observing stations, Earth orientation parameters, precise satellite 
orbits, etc.) for use by a broader scientific community.
• The CDDIS has extensive partnerships through the IAG serving as one of the primary data centers for the geometric 
services and its observing system, GGOS (Global Geodetic Observing System)
. 
What’s Up Next?
• All web activities will transition fall 2016 to HTTPS per U.S. government policy
• HTTPS access to CDDIS archive
 + Archives and users continue to move away from using FTP 
 + Therefore, CDDIS will implement access to it’s full archive through HTTP
 + HTTPS access will continue to use same structure as provided through FTP 
 + HTTPS access is as efficient as FTP transfer without the firewall/router issues of FTP
 + Earthdata login (see left and above) will be used for access through HTTPS
 + FTP access to CDDIS archive will continue but users are encouraged to explore HTTPS capabilities
• Improvements to CDDIS Site Log Viewer
 + Automated ingest to reflect site log updates
 + Link metadata to other applications
• Improvements to CDDIS operations
 + Streamlining archive operations across data types
 + Improved metadata for archive operations and data discovery
NTRIP (real-time GNSS software) client 
interface to CDDIS real-time caster
What’s New?
System Facilities/Architecture Improvements:
• Over the past 7 years CDDIS has experienced double-digit growth culminating in over 1.2B downloads and over 130Tb of data transferred in 2015
• On track to exceed 1.5B files and 180Tb in 2016
• Upgraded hardware procured and installed in new location providing better infrastructure (power, network connectivity, etc.)
 + IT infrastructure designed for 4 “nines” uptime 
 + Multiple redundant 40Gb networks directly connected to the Internet
 + New system implemented with virtual machine architecture for reliability and expandability 
 + Both production and disaster recovery (DR) systems located at different buildings at GSFC
 + Unified storage across both production and DR systems
 + File processing software re-designed for more efficient operations and additional QC
New File Upload Procedure:
• Because of NASA security restrictions, CDDIS can no longer use non-secure FTP for file uploads from data providers
• New, upgraded system was designed to use HTTP protocol for file upload
• Implemented both web and command line interfaces
 + Web interface for simple, interactive uploads
 + Command interface for bulk uploads and scripting; users can make simple modifications to existing scripts for uploads to the new system
• cURL is the supported program for command line access but any program that can do HTTP GET and POST is usable
• Sample code (Java, bash) provided for bulk uploading and scripting
• New system uses EOSDIS Earthdata login 
• Users must first register with EOSDIS to obtain a user ID for access to upload system
• For more information: http://cddis.gsfc.nasa.gov/Data_and_Derived_Products/CDDIS_File_Upload_Documentation.html
Sites streaming real-time
GNSS data thru CDDIS caster
Real-Time GNSS Distribution: 
• CDDIS now operationally streaming real-time GNSS data and derived products in 
support of the IGS Real-Time Service (RTS)
• One of three real-time “casters”
• Data from over 250 global sites and 30 derived product steams
• Expanding to include additional streams
• Real-time system also uses EOSDIS Earthdata login 
• Users must first register with EOSDIS to obtain a user ID for access to real-time 
GNSS caster
Earthdata login 
interface:
Used to register 
and access file 
upload application
(as well as CDDIS
real-time GNSS
streaming system)
File upload 
application
for interactive 
upload of files 
List of real-time 
GNSS data, 
product
streams available 
thru CDDIS 
caster
Current Space Geodesy Site Locations
Abstract: The Crustal Dynamics Data Information System (CDDIS) supports data archiving and distribution activities for the space geodesy and geodynamics community. The main objectives of the system are to store space geodesy and geodynamics related data and products in a central archive, to maintain information about the archival of these data, 
to disseminate these data and information in a timely manner to a global scientific research community, and provide user based tools for the exploration and use of the archive. The CDDIS data system and its archive is a key component in several of the geometric services within the International Association of Geodesy (IAG) and its observing system 
the Global Geodetic Observing System (GGOS), including the IGS, the International DORIS Service (IDS), the International Laser Ranging Service (ILRS), the International VLBI Service for Geodesy and Astrometry (IVS), and the International Earth Rotation and Reference Systems Service (IERS).  The CDDIS provides on-line access to over 17 Tbytes of data 
and derived products in support of the IAG services and GGOS. The system’s archive continues to grow and improve as new activities are supported and enhancements are implemented. Recently, the CDDIS has established a real-time streaming capability for GNSS data and products. Furthermore, enhancements to metadata describing the contents of 
the archive have been developed to facilitate data discovery. This poster will provide a review of the improvements in the system infrastructure that CDDIS has made over the past year for the geodetic community and describe future plans for the system. 
CDDIS: NASA’s Archive of Space Geodesy Data and Products Supporting GGOS Carey Noll/CDDIS Manager (Carey.Noll@nasa.gov)Patrick Michael/Deputy (Patrick.Michael@nasa.gov)NASA Goddard Space Flight Center
Code 61A, Greenbelt, MD 20771, USA
Presented at: 2016 Fall AGU
San Francisco CA, December 12-16, 2016
GGOS Portal and Metadata:
• Requirements/uses:
 + Users
  ► Search/discover for data/products
  ► Find sites/targets of interest
 + Data centers
  ► Organizing/cataloging data/product holdings
  ► Track provenance, usage
  ► Facilitate discovery
• Common to all techniques:
 + Things that observe (STATIONS)
 + Things that are observed (TARGETS)
 + Things that are transmitted/archived (DATA)
 + Things that have been analyzed/are archived (PRODUCTS)
• Each of these “things” have metadata that:
 + Need to be discoverable
 + Need to be searchable
 + Provided to DCs or created by DCs
• Entities flow data/products/information to data centers
• Data centers ingest files/information into archive and extract relevant metadata
• Users retrieve files from archive
• Applications utilize metadata to discover data/products/information
• Data centers transfer metadata to other entities
Supporting GGOS: Metadata
Metadata Developments at CDDIS:
• To aid in the search and discovery process, data must be organized 
and catalogued
• Thus, accurate, complete, and consistent metadata (data about data) 
is a requirement for efficient accessibility
• Metadata describe data attributes:
 + Data acquisition
  ► Information about the target
  ► Information about the station
 + Data quality
  ► Completeness of the data
  ► Gaps in data set
  ► Characteristics that affect reliability of data
 + Data lineage
  ► Provenance
  ► Processing input
  ► Tracking data through transformations/analysis/interpretations
• Non-standard metadata, data type (GNSS, SLR, VLBI, DORIS) specific, 
extracted from incoming files (granules)
• CDDIS currently updating collection and granule level metadata to 
satisfy EOSDIS requirements
 + CDDIS is an EOSDIS DAAC 
 + EOSDIS implementing a “Common Metadata Repository” (CMR)
 + Single source of unified, high-quality, high-performance, and 
reliable Earth Science metadata
 + Metadata can be discovered and accessed through programmatic 
interfaces leveraging standard protocols and APIs
 + CDDIS modifying existing metadata to develop relationships with 
EOSDIS required metadata
Site/Target
Metadata
Content
Metadata
File
Metadata
Categories of CDDIS metadata:
 • File (granule) content (data and derived 
products)
 • File (granule) information (size, source, 
etc.)
 • Site information
 • Target information
Data and products are acquired as part of NASA's Earth Science Data Systems and archived and distributed 
by the Crustal Dynamics Data Information System (CDDIS):
C. Noll, The Crustal Dynamics Data Information System: A resource to support scientific analysis using 
space geodesy, Advances in Space Research,Volume 45, Issue 12, 15 June 2010, Pages 1421-1440, ISSN 
0273-1177, DOI: 10.1016/j.asr.2010.01.018.
The staff welcomes feedback on the CDDIS and in particular the ideas expressed in this poster; contact Carey 
Noll (Carey.Noll@nasa.gov)
Users EOSDIS
GGOS
Search
Engines
M
etadata Transfer
Applications
Archives
Metadata
Databases
Ingest
Product
Files
Data
Files Information
Stations AnalysisCenters
IAG
Services
Processes
Systems
External
CDDIS metadata 
process 
Command line program 
example using cURL for 
scripted uploads
All systems located at NASA Goddard Space Flight Center, Greenbelt MD
https interface to 
CDDIS archive
[/home/user]$ curl -c .urs_cookies -n -L 
http://depot.cddis.eosdis.nasa.gov/CDDIS_F
ileUpload/login/
Welcome to CDDIS File Upload
[/home/user]$ curl -X POST -b .urs_cookies 
-F "fileType=SLR" -F 
"fileContentType=products" -F 
"file[]=@ilrsa.eop.160901.v135.snx.Z" 
http://depot.cddis.eosdis.nasa.gov/CDDIS_F
ileUpload/upload/
Successful upload: 
ilrsa.eop.160901.v135.snx.Z
Successfully uploaded 1 files, out of 1 
attempted
CDDIS
EOSDIS
External
Replicated to
disaster 
recovery site
Ingest EarthdataLogin
Web
Apps
HTTP
Archive
Access
FTP
Archive
Access
CDDIS
Database
(content, file 
metadata)
EOSDIS
CMR
Database
User CommunityData
Providers
Processing
CDDIS
Archive
File metadata, metrics
Content metadata, metrics
Processed data, products
Metadata, metrics
New CDDIS system architecture
Data, product files
Data,
product,
files
GNSS site
SLR site
VLBI site
DORIS site
CDDIS successfully 
transitioned to this new 
system architecture on 
December 01, 2016; 
downtime due to the transfer 
was approximately 30 hours.
https://ntrs.nasa.gov/search.jsp?R=20170004531 2019-08-29T22:43:40+00:00Z


CDDIS: NASA's Archive of Space Geodesy Data and Products Supporting GGOS

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server