Analyzing Big Data Workloads Using Discrete Simulation

Virtualized IT platforms abstracting from physical infrastructure offer easy scaling of storage and compute resources. This is a great progress for handling dynamic big data workloads but holds the danger of solving performance issues just with upscaling, without looking at causes. We propose to analyze the behavior of complex systems handling big data workloads using system modelling and discrete event simulation. It can be very revealing to see a simulated system in action by running simulations in different configurations, showing the impact of a modified system structure on its runtime behavior and resource consumption. With a toolbased system workload simulation example in the Earth Observation data domain we show how this approach can lead to significant resource and cost savings

Innovative Sentinel 5P Level 3 Atmosphere Products: Will Become Accessible Through Innovative Technologies and Interfaces at DLR's Geoservice

The EOC Geoservice (https://geoservice.dlr.de) operationally provides access to innovative Level-3 Products from Copernicus Sentinel 5P, a variety of atmosphere-related Level-3-products from GOME-2/MetOp-A/B/C and a variety of land-surface products such as the world settlement footprint (WSF) and other specific Sentinel-2-products. For more than 10 years, the EOC Geroservice is operational and provides access to all its hosted data collections and products via the OGC compliant interfaces WMS and WCS (data cube). ISO metadata on data collections and products are exposed via compliant catalogue services. In order to better support the currently arising needs for interoperability and for the analysis of long EO timeseries (big data analytics), innovative technologies and interfaces for data discovery, access and analysis are investigated. To facilitate improved metadata discovery, the OpenSearch API and the Spatio Temporal Asset Catalogue (STAC) are both integrated. Globally unique Digital object identifiers (DOIs) for collection metadata are routinely assigned and registered at the responsible DOI Registration Agencies. To enable fast server-based EO big data analytics, OGC’s data analysis and processing API (DAPA) will be integrated into the EOC Geoservice. DAPA brings the algorithms to the data: Algorithms can partly be executed (e. g. on time-slices at the data’s storage location), reducing data transfers. Prerequisite for the efficient combination of different thematic layers in a data cube is the capability to make them accessible on an identical grid: An implementation of OGC’s Discrete Global Grid System (DGGS) is used for this purpose. For all technologies, the GeoServer is the technical backbone. Throughout the above mentioned 10-years-period, DLR has significantly supported its further-development as open-source software. Its most recent improvements (integration of STAC and DAPA interface) have been funded in the framework of the ESA GSTP-project “Technologies for the Management of Long EO Data Time Series” (LOOSE). Integration of all innovative interfaces into an operational data discovery, access and analysis service (EOC Geoservice and DataCube) is supported by the DLR programmatic project "Innovative Produktentwicklung zur Analyse der Atmosphärenzusammensetzung" (INPULS)

Observation of volcanic ash from Puyehue-Cordon Caulle with IASI

On 4 June 2011 an eruption of the Chilean volcano complex Puyehue-Cordon Caulle injected large amounts of volcanic ash into the atmosphere and affected local life as well as hemisphere-wide air traffic. Observations of the Infrared Atmospheric Sounding Interferometer (IASI) flown on board of the MetOp satellite have been exploited to analyze the evolution of the ash plume around the Southern Hemisphere. A novel singular vector-based retrieval methodology, originally developed for observation of desert dust over land and ocean, has been adapted to enable remote sensing of volcanic ash. Since IASI observations in the 8–12 μm window are applied in the retrieval, the method is insensitive to solar illumination and therefore yields twice the observation rate of the ash plume evolution compared to solar backscatter methods from polar orbiting satellites. The retrieval scheme, the emission characteristics and the circumpolar transport of the ash are examined by means of a source-receptor analysis

WDC-RSAT -- Service for Science

Introduction to data access and dissemination services provided by WDC-RSAT as well as its infrastructure

Dicke Luft – Stadtregionen als globale Zentren der Luftverschmutzung

In diesem Beitrag wird der anthropogene Fußabdruck der Luftverschmutzung anhand von satellitengestützten Beobachtungen des Luftschadstoffs NO2 analysiert. Die Betrachtungsebenen umfassen dabei die globale Skala, 30 Megacities und die Megaregion Perlfluss-Delta in China. Dafür werden die Pro-Kopf-Verschmutzung und NO2-Trends quantifiziert, wonach sich die Megacities wirtschaftsräumlich klassifizieren lassen. Megacities des Globalen Südens beispielsweise zeigen bei einer niedrigen Pro-Kopf-Belastung einen durchweg positiven Trend. Am Beispiel der Megaregion Perlfluss-Delta werden erstmalig Satellitendaten des Luftschadstoffs NO2 mit den räumlichen Veränderungen des Siedlungsmusters für den Zeitraum eines Jahrzehnts kombiniert. Es wird gezeigt, dass durch die Verschneidung von NO2 Trends mit dem urbanen Fußabdruck die Analyse räumlich präzisiert werden kann

Derivation of Tropospheric NO2 by Synergistic Use of Satellite Observations and Chemical Transport Modelling

When deriving the NO2 content of the troposphere from satellite borne spectrometers using DOAS two key problems arise: the quantification of the stratospheric contribution and the calculation of the tropospheric air mass factor (hereafter: AMF). This work aims at combining satellite observations and chemical transport modelling to overcome these problems and to gain accurate tropospheric NO2 columns. To take into account the stratospheric (zonal) variability we derive the stratospheric NO2 column at the instrument’s overpass time by means of a stratospheric CTM (ROSE/DLR) driven by meteorological wind- and temperature fields. To avoid a bias, the resulting stratospheric NO2 analysis is scaled to “clean” observation conditions at a reference sector. The stratospheric NO2 slant column density (hereafter: SCD) is derived by applying a geometric AMF. The tropospheric SCD is then derived by simply subtracting the stratospheric SCD from the total SCD derived from SCIAMACHY observations. In order to consider the variability of tropospheric NO2 and to determine a profile dependent tropospheric AMF, the air quality model EURAD-CTM is applied. The tropospheric AMF is derived by weighting the height dependent air mass factors with the relative NO2 concentrations. This allows computing the air mass factor as a function the forecasted NO2 profile shape. In addition, surface albedo, temperature, cloud fraction and cloud top height are considered. Surface reflectivity and the presence of clouds are taken form satellite observations. In an accompanying study we investigated the stratospheric NO2 content using data assimilation and two different MIPAS observational data sets