eddy4R 0.2.0: a DevOps model for community-extensible processing and analysis of eddy-covariance data based on R, Git, Docker, and HDF5

Large differences in instrumentation, site setup, data format, and operating system stymie the adoption of a universal computational environment for processing and analyzing eddy-covariance (EC) data. This results in limited software applicability and extensibility in addition to often substantial inconsistencies in flux estimates. Addressing these concerns, this paper presents the systematic development of portable, reproducible, and extensible EC software achieved by adopting a development and systems operation (DevOps) approach. This software development model is used for the creation of the eddy4R family of EC code packages in the open-source R language for statistical computing. These packages are community developed, iterated via the Git distributed version control system, and wrapped into a portable and reproducible Docker filesystem that is independent of the underlying host operating system. The HDF5 hierarchical data format then provides a streamlined mechanism for highly compressed and fully self-documented data ingest and output. The usefulness of the DevOps approach was evaluated for three test applications. First, the resultant EC processing software was used to analyze standard flux tower data from the first EC instruments installed at a National Ecological Observatory (NEON) field site. Second, through an aircraft test application, we demonstrate the modular extensibility of eddy4R to analyze EC data from other platforms. Third, an intercomparison with commercial-grade software showed excellent agreement (R2  =  1.0 for CO2 flux). In conjunction with this study, a Docker image containing the first two eddy4R packages and an executable example workflow, as well as first NEON EC data products are released publicly. We conclude by describing the work remaining to arrive at the automated generation of science-grade EC fluxes and benefits to the science community at large. This software development model is applicable beyond EC and more generally builds the capacity to deploy complex algorithms developed by scientists in an efficient and scalable manner. In addition, modularity permits meeting project milestones while retaining extensibility with time

NEON’s eddy-covariance: interoperable flux data products, software and services for you, now

Networks of eddy-covariance (EC) towers such as AmeriFlux, ICOS and NEON are vital for providing the necessary distributed observations to address interactions at the soil-vegetation-atmosphere interface. NEON, close to full operation with 47 tower sites, will represent the largest single-provider EC network globally. Its standardized observation and data processing suite is designed specifically for inter-site comparability and analysis of feedbacks across multiple spatial and temporal scales. Furthermore, NEON coordinates EC with rich contextual observations such as airborne remote sensing and in-situ sampling bouts. In January 2018 NEON enters its operational phase, and EC data products, software and services become fully available to the science community at large. These resources strive to incorporate lessons-learned through collaborations with AmeriFlux, ICOS, LTER and others, to suggest novel systemic solutions, and to synergize ongoing research efforts across science communities. Here, we present an overview of the ongoing product release, alongside efforts to integrate and collaborate with existing infrastructures, networks and communities. Near-real-time heat, water and carbon cycle observations in “basic” and “expanded”, self-describing HDF5 formats become accessible from the NEON Data Portal, including an Application Program Interface. Subsequently, they are ingested into the AmeriFlux processing pipeline, together with inclusion in FLUXNET globally harmonized data releases. Software for reproducible, extensible and portable data analysis and science operations management also becomes available. This includes the eddy4R family of R-packages underlying the data product generation, together with the ability to directly participate in open development via GitHub version control and DockerHub image hosting. In addition, templates for science operations management include a web-based field maintenance application and a graphical user interface to simplify problem tracking and resolution along the entire data chain. We hope that this presentation can initiate further collaboration and synergies in challenge areas, and would appreciate input and discussion on continued development. Plain Language Summary For a sustained period of time the eddy-covariance and boundary layer communities have invested technical and scientific expertise into the construction of the National Ecological Observatory Network (NEON). In January 2018 NEON enters its operational phase, and the time has come for our communities to reap the first fruits of their efforts! This presentation intends to create awareness of the resources that become available to our communities: interoperable flux data products, software and assignable asset services. We focus on how these resources will permit to elucidate interactions at the soil-vegetation-atmosphere interface for decades to come: continuous eddy-covariance observations of the surface-atmosphere exchange are tightly coordinated with rich contextual data such as airborne remote sensing and in-situ sampling bouts. In this way new investigative dimensions are provided to capture land-atmosphere feedbacks across multiple spatial and temporal scales

Catalyzing continental-scale carbon cycle science with NEON's first data and software release

Networks of eddy-covariance (EC) towers such as AmeriFlux, ICOS and NEON are vital for providing the necessary distributed observations to address grand challenges in earth system and carbon cycle science. NEON, once fully operational with 47 tower sites, will represent the largest single-provider EC network globally. Its standardized observation and data processing suite is designed specifically for inter-site comparability and analysis of continental-scale ecological change, including rich contextual data such as airborne remote sensing and in-situ sampling bouts. First carbon cycle products become available in 2017, including data and software. These products strive to incorporate lessons-learned through collaborations with AmeriFlux, ICOS, LTER and others, to suggest novel systemic solutions, and to synergize ongoing research efforts across science communities. Here, we present an overview of the ongoing product release, alongside efforts to integrate and synergize with existing infrastructures, networks and communities. Near-real-time carbon cycle observations in “basic” and “expanded”, self-describing HDF5 formats become accessible from the NEON Data Portal, including an Application Program Interface. A pilot project is underway to investigate their subsequent ingest into the AmeriFlux processing pipeline, together with inclusion in FLUXNET globally harmonized data releases. Software for reproducible, extensible and portable data analysis and science operations management also becomes available. This includes the eddy4R family of R-packages underlying the carbon cycle data product generation, together with the ability to directly participate in open development via GitHub version control and Dockerhub image hosting. In addition, templates for science operations management include a web-based field maintenance application and a graphical user interface to simplify problem tracking and resolution along the entire data chain. We hope that this first release of NEON carbon cycle products can initiate further collaboration and synergies in challenge areas, and would appreciate input and discussion on continued development

Measurements of traffic-dominated pollutant emissions in a Chinese megacity

Direct measurements of NOx, CO and aromatic volatile organic compound (VOC) (benzene, toluene, C2-benzenes and C3-benzenes) flux were made for a central area of Beijing using the eddy-covariance technique. Measurements were made during two intensive field campaigns in central Beijing as part of the Air Pollution and Human Health (APHH) project, the first in November–December 2016 and the second during May–June 2017, to contrast wintertime and summertime emission rates. There was little difference in the magnitude of NOx flux between the two seasons (mean NOx flux was 4.41 mg m−2 h−1 in the winter compared to 3.55 mg m−2 h−1in the summer). CO showed greater seasonal variation, with mean CO flux in the winter campaign (34.7 mg m−2 h−1) being over twice that of the summer campaign (15.2 mg m−2 h−1). Larger emissions of aromatic VOCs in summer were attributed to increased evaporation due to higher temperatures. The largest fluxes in NOx and CO generally occurred during the morning and evening rush hour periods, indicating a major traffic source with high midday emissions of CO, indicating an additional influence from cooking fuel. Measured NOx and CO fluxes were then compared to the MEIC 2013 emissions inventory, which was found to significantly overestimate emissions for this region,providing evidence that proxy-based emissions inventories have positive biases in urban centres. This first set of pollutant fluxes measured in Beijing provides an important benchmark of emissions from the city which can help to inform and evaluate current emissions inventories

Eddy covariance measurements highlight sources of nitrogen oxide emissions missing from inventories for central London

During March–June 2017 emissions of nitrogen oxides were measured via eddy covariance at the British Telecom Tower in central London, UK. Through the use of a footprint model the expected emissions were simulated from the spatially resolved National Atmospheric Emissions Inventory for 2017 and compared with the measured emissions. These simulated emissions were shown to underestimate measured emissions during the daytime by a factor of 1.48, but they agreed well overnight. Furthermore, underestimations were spatially mapped, and the areas around the measurement site responsible for differences in measured and simulated emissions were inferred. It was observed that areas of higher traffic, such as major roads near national rail stations, showed the greatest underestimation by the simulated emissions. These discrepancies are partially attributed to a combination of the inventory not fully capturing traffic conditions in central London and both the spatial and temporal resolution of the inventory not fully describing the high heterogeneity of the urban centre. Understanding of this underestimation may be further improved with longer measurement time series to better understand temporal variation and improved temporal scaling factors to better simulate sub-annual emissions

NEONScience/eddy4R: eddy4R-Docker 0.2.0

Accompanying the manuscript "eddy4R 0.2.0: A DevOps model for community-extensible processing and analysis of eddy-covariance data based on R, Git, Docker and HDF5" in Geoscientific Model Development (GMD) http://www.geosci-model-dev-discuss.net/gmd-2016-318/