Comparing the effectiveness of recent algorithms to fill and smooth incomplete and noisy time series

Geophysical time series often feature missing data or data acquired at irregular times. Procedures are needed to either resample these series at systematic time intervals or to generate reasonable estimates at specified times in order to meet specific user requirements or to facilitate subsequent analyses. Interpolation methods have long been used to address this problem, taking into account the fact that available measurements also include errors of measurement or uncertainties. This paper inspects some of the currently used approaches to fill gaps and smooth time series (smoothing splines, Singular Spectrum Analysis and Lomb-Scargle) by comparing their performance in either reconstructing the original record or in minimizing the Mean Absolute Error (MAE) between the underlying model and the available data, using both artificially-generated series or well-known publicly available records. Some methods make no assumption on the type of variability in the data while others hypothesize the presence of at least some dominant frequencies. It will be seen that each method exhibits advantages and drawbacks, and that the choice of an approach largely depends on the properties of the underlying time series and the objective of the research.JRC.H.5-Land Resources Managemen

A roadmap for high-resolution satellite soil moisture applications – confronting product characteristics with user requirements

Soil moisture observations are of broad scientific interest and practical value for a wide range of applications. The scientific community has made significant progress in estimating soil moisture from satellite-based Earth observation data, particularly in operationalizing coarse-resolution (25-50 km) soil moisture products. This review summarizes existing applications of satellite-derived soil moisture products and identifies gaps between the characteristics of currently available soil moisture products and the application requirements from various disciplines. We discuss the efforts devoted to the generation of high-resolution soil moisture products from satellite Synthetic Aperture Radar (SAR) data such as Sentinel-1 C-band backscatter observations and/or through downscaling of existing coarse-resolution microwave soil moisture products. Open issues and future opportunities of satellite-derived soil moisture are discussed, providing guidance for further development of operational soil moisture products and bridging the gap between the soil moisture user and supplier communities

Comparison of the Novel Probabilistic Self-Optimizing Vectorized Earth Observation Retrieval Classifier with Common Machine Learning Algorithms

The Vectorized Earth Observation Retrieval (VEOR) algorithm is a novel algorithm suited to the efficient supervised classification of large Earth Observation (EO) datasets. VEOR addresses shortcomings in well-established machine learning methods with an emphasis on numerical performance. Its characteristics include (1) derivation of classification probability; (2) objective selection of classification features that maximize Cohen’s kappa coefficient (κ) derived from iterative “leave-one-out” cross-validation; (3) reduced sensitivity of the classification results to imbalanced classes; (4) smoothing of the classification probability field to reduce noise/mislabeling; (5) numerically efficient retrieval based on a pre-computed look-up vector (LUV); and (6) separate parametrization of the algorithm for each discrete feature class (e.g., land cover). Within this study, the performance of the VEOR classifier was compared to other commonly used machine learning algorithms: K-nearest neighbors, support vector machines, Gaussian process, decision trees, random forest, artificial neural networks, AdaBoost, Naive Bayes and Quadratic Discriminant Analysis. Firstly, the comparison was performed using synthetic 2D (two-dimensional) datasets featuring different sample sizes, levels of noise (i.e., mislabeling) and class imbalance. Secondly, the same experiments were repeated for 7D datasets consisting of informative, redundant and insignificant features. Ultimately, the benchmarking of the classifiers involved cloud discrimination using MODIS satellite spectral measurements and a reference cloud mask derived from combined CALIOP lidar and CPR radar data. The results revealed that the proposed VEOR algorithm accurately discriminated cloud cover using MODIS data and accurately classified large synthetic datasets with low or moderate levels of noise and class imbalance. On the contrary, VEOR did not feature good classification skills for significantly distorted or for small datasets. Nevertheless, the comparisons performed proved that VEOR was within the 3–4 most accurate classifiers and that it can be applied to large Earth Observation datasets

Comparison of the Novel Probabilistic Self-Optimizing Vectorized Earth Observation Retrieval Classifier with Common Machine Learning Algorithms

The Vectorized Earth Observation Retrieval (VEOR) algorithm is a novel algorithm suited to the efficient supervised classification of large Earth Observation (EO) datasets. VEOR addresses shortcomings in well-established machine learning methods with an emphasis on numerical performance. Its characteristics include (1) derivation of classification probability; (2) objective selection of classification features that maximize Cohen’s kappa coefficient (κ) derived from iterative “leave-one-out” cross-validation; (3) reduced sensitivity of the classification results to imbalanced classes; (4) smoothing of the classification probability field to reduce noise/mislabeling; (5) numerically efficient retrieval based on a pre-computed look-up vector (LUV); and (6) separate parametrization of the algorithm for each discrete feature class (e.g., land cover). Within this study, the performance of the VEOR classifier was compared to other commonly used machine learning algorithms: K-nearest neighbors, support vector machines, Gaussian process, decision trees, random forest, artificial neural networks, AdaBoost, Naive Bayes and Quadratic Discriminant Analysis. Firstly, the comparison was performed using synthetic 2D (two-dimensional) datasets featuring different sample sizes, levels of noise (i.e., mislabeling) and class imbalance. Secondly, the same experiments were repeated for 7D datasets consisting of informative, redundant and insignificant features. Ultimately, the benchmarking of the classifiers involved cloud discrimination using MODIS satellite spectral measurements and a reference cloud mask derived from combined CALIOP lidar and CPR radar data. The results revealed that the proposed VEOR algorithm accurately discriminated cloud cover using MODIS data and accurately classified large synthetic datasets with low or moderate levels of noise and class imbalance. On the contrary, VEOR did not feature good classification skills for significantly distorted or for small datasets. Nevertheless, the comparisons performed proved that VEOR was within the 3–4 most accurate classifiers and that it can be applied to large Earth Observation datasets

Comparing the effectiveness of recent algorithms to fill and smooth incomplete and noisy time series

Geophysical time series often feature missing data or data acquired at irregular times. Procedures are needed to either resample these series at systematic time intervals or to generate reasonable estimates at specified times in order to meet specific user requirements or to facilitate subsequent analyses. Interpolation methods have long been used to address this problem, taking into account the fact that available measurements also include errors of measurement or uncertainties. This paper inspects some of the currently used approaches to fill gaps and smooth time series (smoothing splines, Singular Spectrum Analysis and Lomb-Scargle) by comparing their performance in either reconstructing the original record or in minimizing the Mean Absolute Error (MAE) between the underlying model and the available data, using both artificially-generated series or well-known publicly available records. Some methods make no assumption on the type of variability in the data while others hypothesize the presence of at least some dominant frequencies. It will be seen that each method exhibits advantages and drawbacks, and that the choice of an approach largely depends on the properties of the underlying time series and the objective of the research.JRC.DDG.H.3-Global environement monitorin

Probabilistic approach to cloud and snow detection on Advanced Very High Resolution Radiometer (AVHRR) imagery

Derivation of probability estimates complementary to geophysical data sets has gained special attention over the last years. Information about a confidence level of provided physical quantities is required to construct an error budget of higher-level products and to correctly interpret final results of a particular analysis. Regarding the generation of products based on satellite data a common input consists of a cloud mask which allows discrimination between surface and cloud signals. Further the surface information is divided between snow and snow-free components. At any step of this discrimination process a misclassification in a cloud/snow mask propagates to higher-level products and may alter their usability. Within this scope a novel probabilistic cloud mask (PCM) algorithm suited for the 1 km × 1 km Advanced Very High Resolution Radiometer (AVHRR) data is proposed which provides three types of probability estimates between: cloudy/clear-sky, cloudy/snow and clear-sky/snow conditions. As opposed to the majority of available techniques which are usually based on the decision-tree approach in the PCM algorithm all spectral, angular and ancillary information is used in a single step to retrieve probability estimates from the precomputed look-up tables (LUTs). Moreover, the issue of derivation of a single threshold value for a spectral test was overcome by the concept of multidimensional information space which is divided into small bins by an extensive set of intervals. The discrimination between snow and ice clouds and detection of broken, thin clouds was enhanced by means of the invariant coordinate system (ICS) transformation. The study area covers a wide range of environmental conditions spanning from Iceland through central Europe to northern parts of Africa which exhibit diverse difficulties for cloud/snow masking algorithms. The retrieved PCM cloud classification was compared to the Polar Platform System (PPS) version 2012 and Moderate Resolution Imaging Spectroradiometer (MODIS) collection 6 cloud masks, SYNOP (surface synoptic observations) weather reports, Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations (CALIPSO) vertical feature mask version 3 and to MODIS collection 5 snow mask. The outcomes of conducted analyses proved fine detection skills of the PCM method with results comparable to or better than the reference PPS algorithm

Height-driven linear polarization of the surface emission from quantum dashes

The influence of the nanostructure height on the polarization of the surface emission was systematically investigated for In(Ga) As/InP quantum dashes. Polarization-resolved photoluminescence experiment was compared to theoretical considerations based on the multiband k.p theory and an analytical formula relating the polarization anisotropy to the nano-object geometry was derived. Substantial in-plane structure shape asymmetry induces a pronounced degree of linear polarization in surface emission, which depends strongly not only on the lateral aspect ratio but also on the nanostructure height. Additionally, strongly linearly polarized surface emission (up to 90%) was demonstrated for columnar quantum dashes by combining the in-plane elongation with a significantly increased height.</p