BioClimate: a Science Gateway for Climate Change and Biodiversity research in the EUBrazilCloudConnect project

[EN] Climate and biodiversity systems are closely linked across a wide range of scales. To better understand the mutual interaction between climate change and biodiversity there is a strong need for multidisciplinary skills, scientific tools, and access to a large variety of heterogeneous, often distributed, data sources. Related to that, the EUBrazilCloudConnect project provides a user-oriented research environment built on top of a federated cloud infrastructure across Europe and Brazil, to serve key needs in different scientific domains, which is validated through a set of use cases. Among them, the most data-centric one is focused on climate change and biodiversity research. As part of this use case, the BioClimate Science Gateway has been implemented to provide end-users transparent access to (i) a highly integrated user-friendly environment, (ii) a large variety of data sources, and (iii) different analytics & visualization tools to serve a large spectrum of users needs and requirements. This paper presents a complete overview of BioClimate and the related scientific environment, in particular its Science Gateway, delivered to the end-user community at the end of the project.This work was supported by the EU FP7 EUBrazilCloudConnect Project (Grant Agreement 614048), and CNPq/Brazil (Grant Agreement no 490115/2013-6).Fiore, S.; Elia, D.; Blanquer Espert, I.; Brasileiro, FV.; Nuzzo, A.; Nassisi, P.; Rufino, LAA.... (2019). BioClimate: a Science Gateway for Climate Change and Biodiversity research in the EUBrazilCloudConnect project. Future Generation Computer Systems. 94:895-909. https://doi.org/10.1016/j.future.2017.11.034S8959099

Future of networking is the future of Big Data, The

2019 Summer.Includes bibliographical references.Scientific domains such as Climate Science, High Energy Particle Physics (HEP), Genomics, Biology, and many others are increasingly moving towards data-oriented workflows where each of these communities generates, stores and uses massive datasets that reach into terabytes and petabytes, and projected soon to reach exabytes. These communities are also increasingly moving towards a global collaborative model where scientists routinely exchange a significant amount of data. The sheer volume of data and associated complexities associated with maintaining, transferring, and using them, continue to push the limits of the current technologies in multiple dimensions - storage, analysis, networking, and security. This thesis tackles the networking aspect of big-data science. Networking is the glue that binds all the components of modern scientific workflows, and these communities are becoming increasingly dependent on high-speed, highly reliable networks. The network, as the common layer across big-science communities, provides an ideal place for implementing common services. Big-science applications also need to work closely with the network to ensure optimal usage of resources, intelligent routing of requests, and data. Finally, as more communities move towards data-intensive, connected workflows - adopting a service model where the network provides some of the common services reduces not only application complexity but also the necessity of duplicate implementations. Named Data Networking (NDN) is a new network architecture whose service model aligns better with the needs of these data-oriented applications. NDN's name based paradigm makes it easier to provide intelligent features at the network layer rather than at the application layer. This thesis shows that NDN can push several standard features to the network. This work is the first attempt to apply NDN in the context of large scientific data; in the process, this thesis touches upon scientific data naming, name discovery, real-world deployment of NDN for scientific data, feasibility studies, and the designs of in-network protocols for big-data science

Supporting Climate Research using Named Data Networking

Abstract-Climate and other big data applications face substantial problems in terms of data storage, retrieval, sharing and management. While several community repositories and tools are available to help with climate data, these problems still persist and the community is actively looking for better solutions. In this project we apply NDN to support climate modeling applications. The information-centric nature of NDN, where content becomes a first class entity, simplifies many of the problems in this domain. NDN offers lightweight data publication, discovery and retrieval compared to IP-based solutions. However, introducing a new network architecture to a mature domain that routinely produces petabytes of datasets and a plethora of assorted tools to manipulate them, is a risky proposition. The advantages of NDN alone may not be sufficient to overcome the natural inertia. Our approach is to introduce NDN while carefully avoiding undue disruption to existing workflows. To that extent we employ a user interface that employs familiar filesystem operations to publish, discover and retrieve data, integrated with domain-specific translators that automatically convert and publish datasets as NDN objects. We outline the advantages of NDN in this application domain and the challenges we faced during the adaptation. We believe this is the first exercise in applying NDN in an existing, large, mature application domain

Development of an interface for the conversion of geodata in a NetCDF data model and publication of this data by the use of the web application DChart, related to the CEOP-AEGIS project

The Tibetan Plateau with an extent of about 2,5 million square kilometers at an average altitude higher than 4,700 meters has a significant impact on the Asian monsoon and regulates with its snow and ice reserves the upstream headwaters of seven major south-east Asian rivers. Upon the water supply of these rivers depend over 1,4 billion people, the agriculture, the economics, and the entire ecosystem in this region. As the increasing number of floods and droughts show, these seasonal water reserves however are likely to be influenced by climate change, with negative effects for the downstream water supply and subsequently the food security. The international cooperation project CEOP-AEGIS – funded by the European Commission under the Seventh Framework Program – aims as a result to improve the knowledge of the hydrology and meteorology of the Qinghai-Tibetan Plateau to further understand its role in climate, monsoon and increasing extreme meteorological events. Within the framework of this project, a large variety of earth observation datasets from remote sensing products, model outputs and in-situ ground station measurements are collected and evaluated. Any foreground products of CEOP-AEGIS will have to be made available to the scientific community by an online data repository which is a contribution to the Global Earth Observation System of Systems (GEOSS). The back-end of the CEOP-AEGIS Data Portal relies on a Dapper OPeNDAP web server that serves data stored in the NetCDF file format to a DChart client front-end as web-based user interface. Data from project partners are heterogeneous in its content, and also in its type of storage and metadata description. However NetCDF project output data and metadata has to be standardized and must follow international conventions to achieve a high level of interoperability. Out of these needs, the capabilities of NetCDF, OPeNDAP, Dapper and DChart were profoundly evaluated in order to take correct decisions for implementing a suitable and interoperable NetCDF data model for CEOP-AEGIS data that allows a maximum of compatibility and functionality to OPeNDAP and Dapper / DChart as well. This NetCDF implementation is part of a newly developed upstream data interface that converts and aggregates heterogeneous input data of project partners to standardized NetCDF datasets, so that they can be feed via OPeNDAP to the CEOP-AEGIS Data Portal based on the Dapper / DChart technology. A particular focus in the design of this data interface was set to an intermediate data and metadata representation that easily allows to modify its elements with the scope of achieving standardized NetCDF files in a simple way. Considering the extensive variety and amount of data within this project, it was essential to properly design a data interface that converts heterogeneous input data of project partners to standardized and aggregated NetCDF output files in order to ensure maximum compatibility and functionality within the CEOP-AEGIS Data Portal and subsequently interoperability within the scientific community.:Task of Diploma Thesis ii Declaration of academic honesty vii Abstract ix Acknowledgments xiii Dedication xv Table of Contents xvii List of Figures xxi List of Tables xxiii List of Listings xxv Nomenclature xxvii 1 Introduction 1 1.1 CEOP-AEGIS project . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Objective of this thesis . . . . . . . . . . . . . . . . . . . . . . 8 1.4 Structure of this work . . . . . . . . . . . . . . . . . . . . . . 10 2 Theoretical foundations 13 2.1 NetCDF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.1.1 Data models . . . . . . . . . . . . . . . . . . . . . . . . 16 2.1.2 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.1.3 Dimensions . . . . . . . . . . . . . . . . . . . . . . . . 17 2.1.4 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.1.5 Attributes . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.1.6 NetCDF 3 . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.1.7 NetCDF 4 . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.1.8 Common Data Model . . . . . . . . . . . . . . . . . . . 31 2.1.9 NetCDF libraries and APIs . . . . . . . . . . . . . . . 33 2.1.10 NetCDF utilities . . . . . . . . . . . . . . . . . . . . . 34 2.1.11 NetCDF textual representations . . . . . . . . . . . . . 35 2.1.12 NetCDF conventions . . . . . . . . . . . . . . . . . . . 36 2.2 OPeNDAP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.2.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . 41 2.2.2 OPeNDAP servers . . . . . . . . . . . . . . . . . . . . 42 2.2.3 OPeNDAP clients . . . . . . . . . . . . . . . . . . . . . 47 2.2.4 Data Access Protocol . . . . . . . . . . . . . . . . . . . 48 2.2.5 OPeNDAP data models and data types . . . . . . . . . 49 2.2.6 OPeNDAP and NetCDF . . . . . . . . . . . . . . . . . 53 2.3 Dapper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 2.3.1 Climate Data Portal . . . . . . . . . . . . . . . . . . . 57 2.3.2 System architecture and Dapper services . . . . . . . . 58 2.3.3 Data aggregation . . . . . . . . . . . . . . . . . . . . . 60 2.3.4 Supported conventions of Dapper . . . . . . . . . . . . 61 2.4 DChart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 2.4.1 Design goals . . . . . . . . . . . . . . . . . . . . . . . . 63 2.4.2 Functionality . . . . . . . . . . . . . . . . . . . . . . . 63 2.4.3 System architecture . . . . . . . . . . . . . . . . . . . . 64 2.5 Dapper and DChart configuration . . . . . . . . . . . . . . . . 66 2.5.1 License and release notes . . . . . . . . . . . . . . . . . 67 2.5.2 Dapper and DChart system requirements . . . . . . . . 67 3 Implementation 69 3.1 Scientific data types . . . . . . . . . . . . . . . . . . . . . . . 69 3.1.1 Gridded data . . . . . . . . . . . . . . . . . . . . . . . 70 3.1.2 In-situ data . . . . . . . . . . . . . . . . . . . . . . . . 71 3.2 NetCDF for CEOP-AEGIS . . . . . . . . . . . . . . . . . . . . 71 3.2.1 CF Climate and Forecast Convention . . . . . . . . . . 73 3.2.2 Dapper In-situ Convention . . . . . . . . . . . . . . . . 80 3.2.3 NetCDF implementation for CEOP-AEGIS . . . . . . 89 3.3 CEOP-AEGIS Data Interface . . . . . . . . . . . . . . . . . . 93 3.3.1 Intermediate data model . . . . . . . . . . . . . . . . . 95 3.3.2 Data Interface dependencies . . . . . . . . . . . . . . . 98 3.3.3 Data Interface usage . . . . . . . . . . . . . . . . . . . 98 3.3.4 Data Interface modules . . . . . . . . . . . . . . . . . . 105 3.4 Final products . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 4 Conclusion 111 4.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 4.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 4.3 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 A Appendix 119 A.1 CD-ROM of project data . . . . . . . . . . . . . . . . . . . . . 119 A.2 Flood occurrence maps . . . . . . . . . . . . . . . . . . . . . . 121 A.2.1 Flood occurrence May . . . . . . . . . . . . . . . . . . 122 A.2.2 Flood occurrence August . . . . . . . . . . . . . . . . . 123 A.3 CEOP-AEGIS Data Portal . . . . . . . . . . . . . . . . . . . . 124 A.3.1 Capture image of CEOP-AEGIS Data Portal . . . . . . 125 A.3.2 Dapper configuration file . . . . . . . . . . . . . . . . . 126 A.3.3 DChart configuration file . . . . . . . . . . . . . . . . . 127 A.4 NetCDF data models for CEOP-AEGIS . . . . . . . . . . . . 130 A.4.1 Data model for gridded data . . . . . . . . . . . . . . . 131 A.4.2 Data model for in-situ data . . . . . . . . . . . . . . . 132 A.5 Upstream data interface . . . . . . . . . . . . . . . . . . . . . 133 A.5.1 Data Interface and service chain . . . . . . . . . . . . . 134 A.5.2 Data Interface data flow . . . . . . . . . . . . . . . . . 135 A.5.3 Data Interface data flow 2 . . . . . . . . . . . . . . . . 136 A.5.4 Data Interface modules and classes . . . . . . . . . . . 137 A.5.5 Data Interface NetCDF metadata file for gridded data 138 A.5.6 Data Interface NetCDF metadata file for in-situ data . 139 A.5.7 Data Interface coordinate metadata file for gridded data140 A.5.8 Data Interface coordinate metadata file for in-situ data 140 A.5.9 Data Interface UI main program . . . . . . . . . . . . . 141 A.5.10 Data Interface UI GrADS component . . . . . . . . . . 142 A.5.11 Data Interface UI GDAL component . . . . . . . . . . 143 A.5.12 Data Interface UI CSV component . . . . . . . . . . . 144 A.5.13 Data Interface settings file for gridded data . . . . . . . 145 A.5.14 Data Interface settings file for in-situ data . . . . . . . 146 A.5.15 Data Interface batch file for data conversion via GrADS146 A.5.16 Data Interface batch file for data conversion via GDAL 147 A.5.17 Data Interface batch file for data conversion via CSV . 148 A.6 Pydoc documentation for upstream data interface . . . . . . . 149 A.6.1 grads_2Interface.py . . . . . . . . . . . . . . . . . . . . 150 A.6.2 gdal_2Interface.py . . . . . . . . . . . . . . . . . . . . 155 A.6.3 csv_2Interface.py . . . . . . . . . . . . . . . . . . . . . 162 A.6.4 interface_Main.py . . . . . . . . . . . . . . . . . . . . 167 A.6.5 interface_Settings.py . . . . . . . . . . . . . . . . . . . 172 A.6.6 interface_Control.py . . . . . . . . . . . . . . . . . . . 175 A.6.7 interface_Model.py . . . . . . . . . . . . . . . . . . . . 179 A.6.8 interface_ModelUtilities.py . . . . . . . . . . . . . . . 185 A.6.9 interface_Data.py . . . . . . . . . . . . . . . . . . . . . 189 A.6.10 interface_ProcessingTools.py . . . . . . . . . . . . . . 191 Bibliography 197 Index 205Das Hochplateau von Tibet mit einer Ausdehnung von 2.5 Millionen Quadratkilometer und einer durchschnittlichen Höhe von über 4 700 Meter beeinflusst wesentlich den asiatischen Monsun und reguliert mit seinen Schnee- und Eisreserven den Wasserhaushalt der Oberläufe der sieben wichtigsten Flüsse Südostasiens. Von diesem Wasserzufluss leben 1.4 Milliarden Menschen und hängt neben dem Ackerbau und der Wirtschaft das gesamte Ökosystem in dieser Gegend ab. Wie die zunehmende Zahl an Dürren und Überschwemmungen zeigt, sind diese jahreszeitlich beeinflussten Wasserreserven allen Anscheins nach vom Klimawandel betroffen, mit negativen Auswirkungen für die flussabwärts liegenden Stromgebiete und demzufolge die dortige Nahrungsmittelsicherheit. Das internationale Kooperationsprojekt CEOP-AEGIS – finanziert von der Europäischen Kommission unter dem Siebten Rahmenprogramm – hat sich deshalb zum Ziel gesetzt, die Hydrologie und Meteorologie dieses Hochplateaus weiter zu erforschen, um daraus seine Rolle in Bezug auf das Klima, den Monsun und den zunehmenden extremen Wetterereignissen tiefgreifender verstehen zu können. Im Rahmen dieses Projektes werden verschiedenartigste Erdbeobachtungsdaten von Fernerkundungssystemen, numerischen Simulationen und Bodenstationsmessungen gesammelt und ausgewertet. Sämtliche Endprodukte des CEOP-AEGIS Projektes werden der wissenschaftlichen Gemeinschaft auf Grundlage einer über das Internet erreichbaren Datenbank zugänglich gemacht, welche eine Zuarbeit zur Initiative GEOSS (Global Earth Observing System of Systems) ist. Hintergründig basiert das CEOP-AEGIS Datenportal auf einem Dapper OPeNDAP Internetserver, welcher die im NetCDF Dateiformat gespeicherten Daten der vordergründigen internetbasierten DChart Benutzerschnittstelle auf Grundlage des OPeNDAP Protokolls bereit stellt. Eingangsdaten von Partnern dieses Projektes sind heterogen nicht nur in Bezug ihres Dateninhalts, sondern auch in Anbetracht ihrer Datenhaltung und Metadatenbeschreibung. Die Daten- und Metadatenhaltung der im NetCDF Dateiformat gespeicherten Endprodukte dieses Projektes müssen jedoch auf einer standardisierten Basis internationalen Konventionen folgen, damit ein hoher Grad an Interoperabilität erreicht werden kann. In Anbetracht dieser Qualitätsanforderungen wurden die technischen Möglichkeiten von NetCDF, OPeNDAP, Dapper und DChart in dieser Diplomarbeit gründlich untersucht, damit auf Grundlage dieser Erkenntnisse eine korrekte Entscheidung bezüglich der Implementierung eines für CEOP-AEGIS Daten passenden und interoperablen NetCDF Datenmodels abgeleitet werden kann, das eine maximale Kompatibilität und Funktionalität mit OPeNDAP und Dapper / DChart sicher stellen soll. Diese NetCDF Implementierung ist Bestandteil einer neu entwickelten Datenschnittstelle, welche heterogene Daten von Projektpartnern in standardisierte NetCDF Datensätze konvertiert und aggregiert, sodass diese mittels OPeNDAP dem auf der Dapper / DChart Technologie basierendem Datenportal von CEOP-AEGIS zugeführt werden können. Einen besonderen Schwerpunkt bei der Entwicklung dieser Datenschnittstelle wurde auf eine intermediäre Daten- und Metadatenhaltung gelegt, welche mit der Zielsetzung von geringem Arbeitsaufwand die Modifizierung ihrer Elemente und somit die Erzeugung von standardisierten NetCDF Dateien auf eine einfache Art und Weise erlaubt. In Anbetracht der beträchtlichen und verschiedenartigsten Geodaten dieses Projektes war es schlussendlich wesentlich, eine hochwertige Datenschnittstelle zur Überführung heterogener Eingangsdaten von Projektpartnern in standardisierte und aggregierte NetCDF Ausgansdateien zu entwickeln, um damit eine maximale Kompatibilität und Funktionalität mit dem CEOP-AEGIS Datenportal und daraus folgend ein hohes Maß an Interoperabilität innerhalb der wissenschaftlichen Gemeinschaft erzielen zu können.:Task of Diploma Thesis ii Declaration of academic honesty vii Abstract ix Acknowledgments xiii Dedication xv Table of Contents xvii List of Figures xxi List of Tables xxiii List of Listings xxv Nomenclature xxvii 1 Introduction 1 1.1 CEOP-AEGIS project . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Objective of this thesis . . . . . . . . . . . . . . . . . . . . . . 8 1.4 Structure of this work . . . . . . . . . . . . . . . . . . . . . . 10 2 Theoretical foundations 13 2.1 NetCDF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.1.1 Data models . . . . . . . . . . . . . . . . . . . . . . . . 16 2.1.2 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.1.3 Dimensions . . . . . . . . . . . . . . . . . . . . . . . . 17 2.1.4 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.1.5 Attributes . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.1.6 NetCDF 3 . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.1.7 NetCDF 4 . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.1.8 Common Data Model . . . . . . . . . . . . . . . . . . . 31 2.1.9 NetCDF libraries and APIs . . . . . . . . . . . . . . . 33 2.1.10 NetCDF utilities . . . . . . . . . . . . . . . . . . . . . 34 2.1.11 NetCDF textual representations . . . . . . . . . . . . . 35 2.1.12 NetCDF conventions . . . . . . . . . . . . . . . . . . . 36 2.2 OPeNDAP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.2.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . 41 2.2.2 OPeNDAP servers . . . . . . . . . . . . . . . . . . . . 42 2.2.3 OPeNDAP clients . . . . . . . . . . . . . . . . . . . . . 47 2.2.4 Data Access Protocol . . . . . . . . . . . . . . . . . . . 48 2.2.5 OPeNDAP data models and data types . . . . . . . . . 49 2.2.6 OPeNDAP and NetCDF . . . . . . . . . . . . . . . . . 53 2.3 Dapper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 2.3.1 Climate Data Portal . . . . . . . . . . . . . . . . . . . 57 2.3.2 System architecture and Dapper services . . . . . . . . 58 2.3.3 Data aggregation . . . . . . . . . . . . . . . . . . . . . 60 2.3.4 Supported conventions of Dapper . . . . . . . . . . . . 61 2.4 DChart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 2.4.1 Design goals . . . . . . . . . . . . . . . . . . . . . . . . 63 2.4.2 Functionality . . . . . . . . . . . . . . . . . . . . . . . 63 2.4.3 System architecture . . . . . . . . . . . . . . . . . . . . 64 2.5 Dapper and DChart configuration . . . . . . . . . . . . . . . . 66 2.5.1 License and release notes . . . . . . . . . . . . . . . . . 67 2.5.2 Dapper and DChart system requirements . . . . . . . . 67 3 Implementation 69 3.1 Scientific data types . . . . . . . . . . . . . . . . . . . . . . . 69 3.1.1 Gridded data . . . . . . . . . . . . . . . . . . . . . . . 70 3.1.2 In-situ data . . . . . . . . . . . . . . . . . . . . . . . . 71 3.2 NetCDF for CEOP-AEGIS . . . . . . . . . . . . . . . . . . . . 71 3.2.1 CF Climate and Forecast Convention . . . . . . . . . . 73 3.2.2 Dapper In-situ Convention . . . . . . . . . . . . . . . . 80 3.2.3 NetCDF implementation for CEOP-AEGIS . . . . . . 89 3.3 CEOP-AEGIS Data Interface . . . . . . . . . . . . . . . . . . 93 3.3.1 Intermediate data model . . . . . . . . . . . . . . . . . 95 3.3.2 Data Interface dependencies . . . . . . . . . . . . . . . 98 3.3.3 Data Interface usage . . . . . . . . . . . . . . . . . . . 98 3.3.4 Data Interface modules . . . . . . . . . . . . . . . . . . 105 3.4 Final products . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 4 Conclusion 111 4.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 4.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 4.3 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 A Appendix 119 A.1 CD-ROM of project data . . . . . . . . . . . . . . . . . . . . . 119 A.2 Flood occurrence maps . . . . . . . . . . . . . . . . . . . . . . 121 A.2.1 Flood occurrence May . . . . . . . . . . . . . . . . . . 122 A.2.2 Flood occurrence August . . . . . . . . . . . . . . . . . 123 A.3 CEOP-AEGIS Data Portal . . . . . . . . . . . . . . . . . . . . 124 A.3.1 Capture image of CEOP-AEGIS Data Portal . . . . . . 125 A.3.2 Dapper configuration file . . . . . . . . . . . . . . . . . 126 A.3.3 DChart configuration file . . . . . . . . . . . . . . . . . 127 A.4 NetCDF data models for CEOP-AEGIS . . . . . . . . . . . . 130 A.4.1 Data model for gridded data . . . . . . . . . . . . . . . 131 A.4.2 Data model for in-situ data . . . . . . . . . . . . . . . 132 A.5 Upstream data interface . . . . . . . . . . . . . . . . . . . . . 133 A.5.1 Data Interface and service chain . . . . . . . . . . . . . 134 A.5.2 Data Interface data flow . . . . . . . . . . . . . . . . . 135 A.5.3 Data Interface data flow 2 . . . . . . . . . . . . . . . . 136 A.5.4 Data Interface modules and classes . . . . . . . . . . . 137 A.5.5 Data Interface NetCDF metadata file for gridded data 138 A.5.6 Data Interface NetCDF metadata file for in-situ data . 139 A.5.7 Data Interface coordinate metadata file for gridded data140 A.5.8 Data Interface coordinate metadata file for in-situ data 140 A.5.9 Data Interface UI main program . . . . . . . . . . . . . 141 A.5.10 Data Interface UI GrADS component . . . . . . . . . . 142 A.5.11 Data Interface UI GDAL component . . . . . . . . . . 143 A.5.12 Data Interface UI CSV component . . . . . . . . . . . 144 A.5.13 Data Interface settings file for gridded data . . . . . . . 145 A.5.14 Data Interface settings file for in-situ data . . . . . . . 146 A.5.15 Data Interface batch file for data conversion via GrADS146 A.5.16 Data Interface batch file for data conversion via GDAL 147 A.5.17 Data Interface batch file for data conversion via CSV . 148 A.6 Pydoc documentation for upstream data interface . . . . . . . 149 A.6.1 grads_2Interface.py . . . . . . . . . . . . . . . . . . . . 150 A.6.2 gdal_2Interface.py . . . . . . . . . . . . . . . . . . . . 155 A.6.3 csv_2Interface.py . . . . . . . . . . . . . . . . . . . . . 162 A.6.4 interface_Main.py . . . . . . . . . . . . . . . . . . . . 167 A.6.5 interface_Settings.py . . . . . . . . . . . . . . . . . . . 172 A.6.6 interface_Control.py . . . . . . . . . . . . . . . . . . . 175 A.6.7 interface_Model.py . . . . . . . . . . . . . . . . . . . . 179 A.6.8 interface_ModelUtilities.py . . . . . . . . . . . . . . . 185 A.6.9 interface_Data.py . . . . . . . . . . . . . . . . . . . . . 189 A.6.10 interface_ProcessingTools.py . . . . . . . . . . . . . . 191 Bibliography 197 Index 20

Environmental Observing Systems for Assessing Impacts of Land Use Change

The purpose of this research is to evaluate the application of technology solutions for enabling environmental research. This project develops a complete environmental observing system in support of an investigation of the hydrological and ecological impacts of land use change on the coast of South Carolina. The land use change study is an ongoing multi-disciplinary effort involving the collection of a large number of monitored and sampled parameters at Bannockburn Plantation, which is located near the City of Georgetown in South Carolina. Long term monitoring will support a hydrological and ecological assessment of the study site before, during, and after proposed residential and commercial development. There are three primary objectives of this work. The first is the comprehensive instrumentation of a portion of the Bannockburn study site for hydrological parameters. The second is the implementation of a wireless sensor network to support the remote acquisition of monitored data. The third objective is the creation of a reliable and robust software solution for transmitting and distributing real time observation data. Collectively, these three objectives provide a complete environmental observing system. Results of this work will benefit land use change research by providing access to real time observation data and enabling the integration of powerful analysis tools. This research will provide a basis for the development of future environmental observation systems

Helmholtz Portfolio Theme Large-Scale Data Management and Analysis (LSDMA)

The Helmholtz Association funded the "Large-Scale Data Management and Analysis" portfolio theme from 2012-2016. Four Helmholtz centres, six universities and another research institution in Germany joined to enable data-intensive science by optimising data life cycles in selected scientific communities. In our Data Life cycle Labs, data experts performed joint R&D together with scientific communities. The Data Services Integration Team focused on generic solutions applied by several communities

Earth Observation Open Science and Innovation

geospatial analytics; social observatory; big earth data; open data; citizen science; open innovation; earth system science; crowdsourced geospatial data; citizen science; science in society; data scienc

Improving the Physical Processes and Model Integration Functionality of an Energy Balance Model for Snow and Glacier Melt

The Hindu-Kush Himalayan region possesses a large resource of snow and ice, which acts as a freshwater reservoir for irrigation, domestic water consumption or hydroelectric power for billions of people in South Asia. Monitoring hydrologic resources in this region is challenging because of the difficulty of installing and maintaining a climate and hydrologic monitoring network, limited transportation and communication infrastructure and difficult access to glaciers. As a result of the high, rugged topographic relief, ground observations in the region are extremely sparse. Reanalysis data offer the potential to compensate for the data scarcity, which is a barrier in hydrological modeling and analysis for improving water resources management. Reanalysis weather data products integrate observations with atmospheric model physics to produce a spatially and temporally complete weather record in the post-satellite era. This dissertation creates an integrated hydrologic modeling system that tests whether streamflow prediction can be improved by taking advantage of the National Aeronautics and Space Administration (NASA) remote sensing and reanalysis weather data products in physically based energy balance snow melt and hydrologic models. This study also enhances the energy balance snowmelt model by adding capability to quantify glacier melt. The novelty of this integrated modeling tool resides in allowing the user to isolate various components of surface water inputs (rainfall, snow and glacier ice melt) in a cost-free, open source graphical-user interface-based system that can be used for government and institutional decision-making. Direct, physically based validation of this system is challenging due to the data scarcity in this region, but, to the extent possible, the model was validated through comparison to observed streamflow and to point measurements at locations in the United States having available dat