Fast regridding of large, complex geospatial datasets

In the earth sciences, data are commonly cast on complex grids in order to model irregular domains such as coastlines, or to evenly distribute grid points over the globe. It is common for a scientist to wish to re-cast such data onto a grid that is more amenable to manipulation, visualization, or comparison with other data sources. The complexity of the grids presents a significant technical difficulty to the regridding process. In particular, the regridding of complex grids may suffer from severe performance issues, in the worst case scaling with the product of the sizes of the source and destination grids. We present a mechanism for the fast regridding of such datasets, based upon the construction of a spatial index that allows fast searching of the source grid. We discover that the most efficient spatial index under test (in terms of memory usage and query time) is a simple look-up table. A kd-tree implementation was found to be faster to build and to give similar query performance at the expense of a larger memory footprint. Using our approach, we demonstrate that regridding of complex data may proceed at speeds sufficient to permit regridding on-the-fly in an interactive visualization application, or in a Web Map Service implementation. For large datasets with complex grids the new mechanism is shown to significantly outperform algorithms used in many scientific visualization packages

Fast regridding of large, complex geospatial datasets

Fast regridding of large, comple

A web map service implementation for the visualization of multidimensional gridded environmental data

We describe ncWMS, an implementation of the Open Geospatial Consortium's Web Map Service (WMS) specification for multidimensional gridded environmental data. ncWMS can read data in a large number of common scientific data formats – notably the NetCDF format with the Climate and Forecast conventions – then efficiently generate map imagery in thousands of different coordinate reference systems. It is designed to require minimal configuration from the system administrator and, when used in conjunction with a suitable client tool, provides end users with an interactive means for visualizing data without the need to download large files or interpret complex metadata. It is also used as a “bridging” tool providing interoperability between the environmental science community and users of geographic information systems. ncWMS implements a number of extensions to the WMS standard in order to fulfil some common scientific requirements, including the ability to generate plots representing timeseries and vertical sections. We discuss these extensions and their impact upon present and future interoperability. We discuss the conceptual mapping between the WMS data model and the data models used by gridded data formats, highlighting areas in which the mapping is incomplete or ambiguous. We discuss the architecture of the system and particular technical innovations of note, including the algorithms used for fast data reading and image generation. ncWMS has been widely adopted within the environmental data community and we discuss some of the ways in which the software is integrated within data infrastructures and portals

A Web Map Service implementation for the visualization of multidimensional gridded environmental data

We describe ncWMS, an implementation of the Open Geospatial Consortium’s Web Map Service (WMS) specification for multidimensional gridded environmental data. ncWMS can read data in a large number of common scientific data formats – notably the NetCDF format with the Climate and Forecast conventions – then efficiently generate map imagery in thousands of different coordinate reference systems. It is designed to require minimal configuration from the system administrator and, when used in conjunction with a suitable client tool, provides end users with an interactive means for visualizing data without the need to download large files or interpret complex metadata. It is also used as a “bridging” tool providing interoperability between the environmental science community and users of geographic information systems. ncWMS implements a number of extensions to the WMS standard in order to fulfil some common scientific requirements, including the ability to generate plots representing timeseries and vertical sections. We discuss these extensions and their impact upon present and future interoperability. We discuss the conceptual mapping between the WMS data model and the data models used by gridded data formats, highlighting areas in which the mapping is incomplete or ambiguous. We discuss the architecture of the system and particular technical innovations of note, including the algorithms used for fast data reading and image generation. ncWMS has been widely adopted within the environmental data community and we discuss some of the ways in which the software is integrated within data infrastructures and portals

Statistical analysis of systematic differences in the calculated pollutant concentrations of the models ECMWF/CAMS (regional reanalysis) and Polyphemus/ DLR

In the last two decades, air pollution was viewed as a very serious issue due to the development of infrastructure all over the world. Environmental stressors such as air temperature, radiation, humidity, wind, noise, pollens, and air pollutants (e.g., O3, NO2, PM10, PM2.5) can affect human health in a variety of ways. With the Copernicus Atmospheric Monitoring Service (CAMS) and the air quality in-situ measurements from the European Environmental Agency, a wealth of data of unprecedented quality and spatiotemporal resolution are available. These data are supplemented by available spatiotemporal high-resolution numerical models like chemical-transport models for the comprehensive description of the environmental conditions. Their advantages are constant coverage and high spatial and temporal resolution. However, it is very important to assess the model performances and comparability with in-situ or satellite observations. The main focus of this paper is to perform a comparison of the outputs of the Copernicus Atmosphere Monitoring Service (CAMS) – Europe Air Quality Reanalysis data and the chemical transport model POLYPHEMUS/DLR, with in-situ measurements (station data). The scope is to assess the discrepancies concerning the different chemical species and to provide statistical indicators like Mean Bias, FGE, RMSE, and Trend Analysis and correction weights describing the different characteristics of the models. Also, a Machine Learning approach was applied as an exploratory task, with the goal to predict concentrations at in-situ stations and to identify the influence of each parameters considered by the Polyphemus model. From the results, it was found that Polyphemus/ DLR model overestimates NO2, PM2.5, and PM10 and underestimates the O3, concentrations in urban and rural areas over the time window considered [June 2016 to Dec 2018]. CAMS outputs especially for PM10 and PM2.5 deviates from station observations though the outputs are corrected using EEA air quality station datasets. Overall, the parameters like surface temperature, boundary layer height and season were found to play a major role in both urban and rural regions. There are also significant changes in the influence of some parameters depending on location. This comparison study will help to understand the model performances (overestimation and underestimation) for each of the pollutants and help to select modelled data for health and air pollution-related research in the future