Effects of soil sample pretreatments and standardised rewetting as interacted with sand classes on Vis-NIR predictions of clay and soil organic carbon

Numerous studies have examined the soil analytical potential of diffuse reflectance spectroscopy in the near infrared range, alone or combined with the visible range (Vis-NIR). Soil organic matter (SOM), soil organic carbon (SOC) and clay content are the most commonly and successfully predicted parameters, but predictions are quite variable due e.g. to the range of soil types covered by the calibrations. Especially organic matter predictions are also suggested to be influenced by for example soil moisture content and inclusion of the visible range in the calibration. Excess quartz sand is also suggested to have a negative influence. This study was undertaken to examine the effect of a selection of standardised sample pretreatment procedures, including rewetting, on predictions of clay and SOC content. A subset of 400 samples was selected from a dataset of 3000 Swedish agricultural soils to cover clay and organic matter contents without co-variation. The selected samples were analysed by NIR and Vis-NIR on air-dry samples, either carefully mixed to avoid stratification of particle size classes or shaken to promote separation, resulting in predominantly larger particles being analysed. Unshaken samples were also analysed immediately after standardised additional drying at 35°C for 12 hours and stepwise volumetric rewetting up to 30%. Shaking and additional drying had small negative effects on clay predictions, while drying only had small positive effects on SOC predictions. Volumetric rewetting to 20 or 30% before scanning reduced clay prediction errors by up to 15%, RMSEP reduced from 5.4 % clay to 4.5 % clay, and SOC prediction errors by up to 30%, from 0.9 % SOC to 0.6 % SOC, indicating that standardised rewetting should be considered. The mechanisms concerned could not be specifically identified, but known bands for water, hydroxyl and clay mineral-dependent absorption near 1400, 1900 and 2200 nm were involved in the improved clay calibrations and bands near 1700, 2000, 2300 and 2350 nm in the improved SOC calibrations. The SOC predictions were most inaccurate for soils with a high sand content. For these samples the average prediction error was more than twice as high as those for less sandy samples. Rewetting eliminated this bias, largely explaining the positive effects of rewetting

Influence of the number of samples on prediction error of visible and near infrared spectroscopy of selected soil properties at the farm scale.

Although visible and near infrared (vis-NIR) spectroscopy has proved to be a fast, inexpensive and relatively accurate tool to measure soil properties, considerable research is required to optimise the calibration procedure and establish robust calibration models. This paper reports on the influence of the number of samples used for the development of farm-scale calibration models for moisture content (MC), total nitrogen (TN) and organic carbon (OC) on the prediction error expressed as root mean square error of prediction (RMSEP). Fresh (wet) soil samples collected from four farms in Czech Republic, Germany, Denmark and the UK were scanned with a fibre type vis-NIR, AgroSpec spectrophotometer (tec5 Technology for Spectroscopy, Germany) with a spectral range of 305 - 2200 nm. Spectra were divided into calibration (two-third) and prediction (one-third) sets and the calibration spectra were subjected to a partial least squares regression (PLSR) with leave-one-out cross validation using Unscrambler 7.8 software (Camo Inc., Oslo, Norway). The RMSEP values of models with large sample number (46 - 84 samples from each farm) were compared with those of models developed using small sample number (25 samples selected from the large sample set of each farm) for the same variation range. Both large set and small set models were validated by the same prediction set for each property. Further PLSR analysis was carried out on samples from the German farm, with different sample number of the calibration set of 25, 50, 75 and 100 samples. Results showed that the large-size dataset models resulted in lower RMSEP values than the small-size dataset models for all the soil properties studied. The results also demonstrated that with the increase in sample number used in the calibration set, RMSEP decreased in almost linear fashion, although the largest decrease was between 25 and 50 samples. Therefore, it is recommended to chose the number of samples according to accuracy required, although 50 soil samples is considered appropriate in this study to establish calibration models of TN, OC and MC with smaller expected prediction errors as compared with smaller sample numbers

Comparison between random forests, artificial neural networks and gradient boosted machines methods of on-line vis-NIR spectroscopy measurements of soil total nitrogen and total carbon

Accurate and detailed spatial soil information about within-field variability is essential for variable-rate applications of farm resources. Soil total nitrogen (TN) and total carbon (TC) are important fertility parameters that can be measured with on-line (mobile) visible and near infrared (vis-NIR) spectroscopy. This study compares the performance of local farm scale calibrations with those based on the spiking of selected local samples from both fields into an European dataset for TN and TC estimation using three modelling techniques, namely gradient boosted machines (GBM), artificial neural networks (ANNs) and random forests (RF). The on-line measurements were carried out using a mobile, fiber type, vis-NIR spectrophotometer (305-2200 nm) (AgroSpec from tec5, Germany), during which soil spectra were recorded in diffuse reflectance mode from two fields in the UK. After spectra pre-processing, the entire datasets were then divided into calibration (75%) and prediction (25%) sets, and calibration models for TN and TC were developed using GBM, ANN and RF with leave-one-out cross-validation. Results of cross-validation showed that the effect of spiking of local samples collected from a field into an European dataset when combined with RF has resulted in the highest coefficients of determination (R-2) values of 0.97 and 0.98, the lowest root mean square error (RMSE) of 0.01% and 0.10%, and the highest residual prediction deviations (RPD) of 5.58 and 7.54, for TN and TC, respectively. Results for laboratory and on-line predictions generally followed the same trend as for cross-validation in one field, where the spiked European dataset-based RF calibration models outperformed the corresponding GBM and ANN models. In the second field ANN has replaced RF in being the best performing. However, the local field calibrations provided lower R-2 and RPD in most cases. Therefore, from a cost-effective point of view, it is recommended to adopt the spiked European dataset-based RF/ANN calibration models for successful prediction of TN and TC under on-line measurement conditions

Online measurement of soil organic carbon as correlated with wheat normalised difference vegetation index in a vertisol field

This study explores the potential of visible and near infrared (vis-NIR) spectroscopy for onlinemeasurement of soil organic carbon (SOC). It also attempts to explore correlations and similarities between the spatial distribution of SOC and normalized differential vegetation index (NDVI) of a wheat crop. The online measurement was carried out in a clay vertisol field covering 10 ha of area in Karacabey, Bursa, Turkey. Kappa statistics were carried out between different SOC and NDVI data to investigate potential similarities. Calibration model of SOC in full cross-validationresulted in a good accuracy ( 2 = 0.75, root mean squares error of prediction (RMSEP) = 0.17%, and ratio of prediction deviation (RPD) = 1.81). The validation of the calibration model using laboratory spectra provided comparatively better prediction accuracy ( 2 = 0.70, RMSEP = 0.15%, and RPD = 1.78), as compared to the online measured spectra ( 2 = 0.60, RMSEP = 0.20%, and RPD = 1.41). Although visual similarity was clear, low similarity indicated by a lowKappa value of 0.259 was observed between the online vis-NIR predicted full-point (based on all pointsmeasured in the field, e.g., 6486 points)map of SOC and NDVI map

Predictive performance of mobile vis-near infrared spectroscopy for key soil properties at different geographical scales by using spiking and data mining techniques

The development of accurate visible and near infrared (vis-NIR) spectroscopy calibration models for selected soil properties based on mobile measurements is essential for site specific soil management at fine sampling scale. The objective of the present study was to compare the mobile and laboratory prediction performance of vis-NIR spectroscopy for total nitrogen (TN), total carbon (TC) and soil moisture content (MC) of field soil samples based on single field (SFD), two-field dataset (TFD), UK national dataset (UND) and European continental dataset (ECD) calibration models developed with linear and nonlinear data mining techniques including spiking. Fresh soil samples collected from fields in the UK, Czech Republic, Germany, Denmark and the Netherlands were scanned with a fibre-type vis-NIR spectrophotometer (tec5 Technology for Spectroscopy, Germany), with a spectral range of 305–2200 nm. After dividing spectra into calibration (75%) and validation (25%) sets, spectra in the calibration set were subjected to three multivariate calibration models, including the partial least squares regression (PLSR), multivariate adaptive regression splines (MARS) and support vector machines (SVM), with leave-one-out cross-validation to establish calibration models of TN, TC and MC. Results showed that the best model performance in cross-validation was obtained with MARS methods for the majority of dataset scales used, whereas the lowest model performance was obtained with the SFD. The effect of spiking was significant and the best model performance in general term was obtained when local samples collected from two target fields in the UK were spiked with the ECD, with coefficients of determination (R2) values of 0.96, 0.98 and 0.93, root mean square error (RMSE) of 0.01, 0.1 and 1.75, and ratio of performance to interquartile distance (RPIQ) of 7.46, 6.57 and 3.98, for TC, TN and MC, respectively. Therefore, these results suggest that ECD vis-NIR MARS calibration models can be successfully used to predict TN, TC and MC under both laboratory and mobile measurement conditions

The Brazilian Soil Spectral Service (BraSpecS): A User-Friendly System for Global Soil Spectra Communication

Although many Soil Spectral Libraries (SSLs) have been created globally, these libraries still have not been operationalized for end-users. To address this limitation, this study created an online Brazilian Soil Spectral Service (BraSpecS). The system was based on the Brazilian Soil Spectral Library (BSSL) with samples collected in the Visible–Near–Short-wave infrared (vis–NIR–SWIR) and Midinfrared (MIR) ranges. The interactive platform allows users to find spectra, act as custodians of the data, and estimate several soil properties and classification. The system was tested by 500 Brazilian and 65 international users. Users accessed the platform (besbbr.com.br), uploaded their spectra, and received soil organic carbon (SOC) and clay content prediction results via email. The BraSpecS prediction provided good results for Brazilian data, but performed variably for other countries. Prediction for countries outside of Brazil using local spectra (External Country Soil Spectral Libraries, ExCSSL) mostly showed greater performance than BraSpecS. Clay R2 ranged from 0.5 (BraSpecS) to 0.8 (ExCSSL) in vis–NIR–SWIR, but BraSpecS MIR models were more accurate in most situations. The development of external models based on the fusion of local samples with BSSL formed the Global Soil Spectral Library (GSSL). The GSSL models improved soil properties prediction for different countries. Nevertheless, the proposed system needs to be continually updated with new spectra so they can be applied broadly. Accordingly, the online system is dynamic, users can contribute their data and the models will adapt to local information. Our community-driven web platform allows users to predict soil attributes without learning soil spectral modeling, which will invite end-users to utilize this powerful technique

Rapid detection of soil carbonates by means of NIR spectroscopy, deep learning methods and phase quantification by powder Xray diffraction

Soil NIR spectral absorbance/reflectance libraries are utilized towards improving agricultural production and analysis of soil properties which are key prerequisite for agroecological balance and environmental sustainability. Carbonates in particular, represent a soil property which is mostly affected even by mild, let alone extreme, changes of environmental conditions during climate change. In this study we propose a rapid and efficient way to predict carbonates content in soil by means of FT NIR reflectance spectroscopy and by use of deep learning methods. We exploited multiple machine learning methods, such as: 1) a MLP Regressor and 2) a CNN and compare their performance with other traditional ML algorithms such as PLSR, Cubist and SVM on the combined dataset of two NIR spectral libraries: KSSL (USDA), a dataset of soil samples reflectance spectra collected nationwide, and LUCAS TopSoil (European Soil Library) which contains soil sample absorbance spectra from all over the European Union, and use them to predict carbonate content on never before seen soil samples. Soil samples in KSSL and in TopSoil spectral libraries were acquired in the spectral region of visNIR, however in this study, only the NIR spectral region was utilized. Quantification of carbonates by means of Xray Diffraction is in good agreement with the volumetric method and the MLP prediction. Our work contributes to rapid carbonates content prediction in soil samples in cases where: 1) no volumetric method is available and 2) only NIR spectra absorbance data are available. Up till now and to the best of our knowledge, there exists no other study, that presents a prediction model trained on such an extensive dataset with such promising results on unseen data, undoubtedly supporting the notion that deep learning models present excellent prediction tools for soil carbonates content.Comment: 39 pages, 5 figure

Predicting soil organic carbon in a small farm system using in situ spectral measurements and the random forest regression

A research report submitted to the Faculty of Science, University of the Witwatersrand, Johannesburg, in partial fulfillment of the requirements for the degree of Master of Science (Geographical Information Sciences and Remote Sensing) Johannesburg, 2017Soil organic carbon is considered as the most determining indicator of soil fertility. The purpose of this research was to predict the soil organic carbon in the Mokhotlong region, eastern of Lesotho using in situ spectral measurements and random forest regression. Soil reflectance spectra were acquired by a portable field spectrometer. The performance of random forest regression was assessed by comparing it with one of the most popular models in spectroscopy, partial least square regression. Laboratory spectroscopy measurements of the soil samples were analysed for assessing the accuracy of in situ spectroscopy based-models. The effect of the Savitzky−Golay first derivative in improving partial least square regression and random forest regression in both spectral data was also assessed. The results indicated that the random forest regression could accurately predict the soil organic carbon contents on an independent dataset using in situ spectroscopy data (RPD = 3.77, Rp2= 0.88, RMSEP = 0.64%). The overall best predictive model was achieved with the derivative laboratory spectral data using random forest with the optimum number of key wavelengths (RPD = 3.77, Rp2= 0.88, RMSEP = 0.64%). In contrast, partial least square regression was likely to overfit the calibration dataset. Important wavelengths to predict soil organic contents were localised around the visible range (400-700 nm). An implication of this research is that soil organic carbon can accurately be estimated using derivative in situ spectroscopy measurements and random forest regression with key wavelengths.MT 201

Novel applications of spectroscopy to characterize soil variation

This thesis embodies a collection of novel studies related to the use of multivariate information provided by spectroscopic tools such as Visible and Near Infrared (Vis-NIR) spectrometers to represent soil variation. The general structure is organized following the increasing levels of soil complexity, starting from the characterization of soil aggregates and the identification of soil colloids, to the recognition of soil horizons and their boundaries in the soil profile, to finally the depiction of soil type’s distribution in the landscape. Briefly, Chapter 1 is written as a rationale, emphasising the need for up-to-date methodologies for making effective use of the increasing amount of soil information produced worldwide. Chapter 2 presents the development of a new methodology for the measure of soil aggregate stability and the further use of spectroscopic information to predict its values. Chapter 3 gives examples of the use of Vis-NIR spectral libraries for the prediction of soil properties. Chapter 4 presents the development of a new method for the identification of soil horizons and their boundaries using fuzzy clustering of Vis-NIR spectra. Chapter 5 expands into a new way of measuring the diversity of soils into the landscape, introducing two new indices for measuring soil diversity or “Functional Pedodiversity” inspired in previous studies in Functional Ecology. Finally Chapter 6 discusses the main findings of this thesis and foresees issues, challenges and opportunities in the area of spectroscopy and multivariate soil data analysis