Comparison between random forests, artificial neural networks and gradient boosted machines methods of on-line vis-NIR spectroscopy measurements of soil total nitrogen and total carbon

Accurate and detailed spatial soil information about within-field variability is essential for variable-rate applications of farm resources. Soil total nitrogen (TN) and total carbon (TC) are important fertility parameters that can be measured with on-line (mobile) visible and near infrared (vis-NIR) spectroscopy. This study compares the performance of local farm scale calibrations with those based on the spiking of selected local samples from both fields into an European dataset for TN and TC estimation using three modelling techniques, namely gradient boosted machines (GBM), artificial neural networks (ANNs) and random forests (RF). The on-line measurements were carried out using a mobile, fiber type, vis-NIR spectrophotometer (305-2200 nm) (AgroSpec from tec5, Germany), during which soil spectra were recorded in diffuse reflectance mode from two fields in the UK. After spectra pre-processing, the entire datasets were then divided into calibration (75%) and prediction (25%) sets, and calibration models for TN and TC were developed using GBM, ANN and RF with leave-one-out cross-validation. Results of cross-validation showed that the effect of spiking of local samples collected from a field into an European dataset when combined with RF has resulted in the highest coefficients of determination (R-2) values of 0.97 and 0.98, the lowest root mean square error (RMSE) of 0.01% and 0.10%, and the highest residual prediction deviations (RPD) of 5.58 and 7.54, for TN and TC, respectively. Results for laboratory and on-line predictions generally followed the same trend as for cross-validation in one field, where the spiked European dataset-based RF calibration models outperformed the corresponding GBM and ANN models. In the second field ANN has replaced RF in being the best performing. However, the local field calibrations provided lower R-2 and RPD in most cases. Therefore, from a cost-effective point of view, it is recommended to adopt the spiked European dataset-based RF/ANN calibration models for successful prediction of TN and TC under on-line measurement conditions

A Comparison of VNIR and MIR Spectroscopy for Predicting Various Soil Properties

Soil plays an important role in our daily lives, namely producing food, cleaning water and storing carbon. The ability to rapidly and cost-effectively quantify the various components of soils can help us understand and better manage this important resource. This study aims to compare the ability of visible near-infrared (VNIR) spectroscopy and mid-infrared (MIR) spectroscopy to quickly and accurately predict various important soil properties (electrical conductivity, soil pH, cation exchange capacity, exchangeable cations, phosphorus, carbon, beta-glucosidase enzyme activity and nitrogen). Prediction models were developed using partial least squares regression (PLSR) techniques. Three different calibration sampling methods were tested along with various spectral preprocessing techniques to find the best predictive ability of VNIR and MIR. Soil components related to carbon, nitrogen, and cation exchange capacity had good predictive ability (R2 \u3e 0.8) by both VNIR and MIR, but MIR was more accurate. Electrical conductivity, sodium cations, and phosphorus were poorly predicted by both (\u3c0.71). VNIR models were not as robust as MIR models but could be potentially useful for qualitative analyses when rapid analyses are preferred over methods are more accurate. MIR predictions overall yielded more accurate predictions than VNIR and could potentially be used as a surrogate method for timely laboratory techniques for spectrally active soil components. Advisor: Paul Hanso

Comparison of Hyperspectral Imaging and Near-Infrared Spectroscopy to Determine Nitrogen and Carbon Concentrations in Wheat

Hyperspectral imaging (HSI) is an emerging rapid and non-destructive technology that has promising application within feed mills and processing plants in poultry and other intensive animal industries. HSI may be advantageous over near infrared spectroscopy (NIRS) as it scans entire samples, which enables compositional gradients and sample heterogenicity to be visualised and analysed. This study was a preliminary investigation to compare the performance of HSI with that of NIRS for quality measurements of ground samples of Australian wheat and to identify the most important spectral regions for predicting carbon (C) and nitrogen (N) concentrations. In total, 69 samples were scanned using an NIRS (400–2500 nm), and two HSI cameras operated in 400–1000 nm (VNIR) and 1000–2500 nm (SWIR) spectral regions. Partial least square regression (PLSR) models were used to correlate C and N concentrations of 63 calibration samples with their spectral reflectance, with 6 additional samples used for testing the models. The accuracy of the HSI predictions (full spectra) were similar or slightly higher than those of NIRS (NIRS Rc2 for C = 0.90 and N = 0.96 vs. HSI Rc2 for C (VNIR) = 0.97 and N (SWIR) = 0.97). The most important spectral region for C prediction identified using HSI reflectance was 400–550 nm with R2 of 0.93 and RMSE of 0.17% in the calibration set and R2 of 0.86, RMSE of 0.21% and ratio of performance to deviation (RPD) of 2.03 in the test set. The most important spectral regions for predicting N concentrations in the feed samples included 1451–1600 nm, 1901–2050 nm and 2051–2200 nm, providing prediction with R2 ranging from 0.91 to 0.93, RMSE ranging from 0.06% to 0.07% in the calibration sets, R2 from 0.96 to 0.99, RMSE of 0.06% and RPD from 3.47 to 3.92 in the test sets. The prediction accuracy of HSI and NIRS were comparable possibly due to the larger statistical population (larger number of pixels) that HSI provided, despite the fact that HSI had smaller spectral range compared with that of NIRS. In addition, HSI enabled visualising the variability of C and N in the samples. Therefore, HSI is advantageous compared to NIRS as it is a multifunctional tool that poses many potential applications in data collection and quality assurance within feed mills and poultry processing plants. The ability to more accurately measure and visualise the properties of feed ingredients has potential economic benefits and therefore additional investigation and development of HSI in this application is warranted

HIRIS (High-Resolution Imaging Spectrometer: Science opportunities for the 1990s. Earth observing system. Volume 2C: Instrument panel report

The high-resolution imaging spectrometer (HIRIS) is an Earth Observing System (EOS) sensor developed for high spatial and spectral resolution. It can acquire more information in the 0.4 to 2.5 micrometer spectral region than any other sensor yet envisioned. Its capability for critical sampling at high spatial resolution makes it an ideal complement to the MODIS (moderate-resolution imaging spectrometer) and HMMR (high-resolution multifrequency microwave radiometer), lower resolution sensors designed for repetitive coverage. With HIRIS it is possible to observe transient processes in a multistage remote sensing strategy for Earth observations on a global scale. The objectives, science requirements, and current sensor design of the HIRIS are discussed along with the synergism of the sensor with other EOS instruments and data handling and processing requirements

Using a VNIR Spectral Library to Model Soil Carbon and Total Nitrogen Content

n-situ soil sensor systems based on visible and near infrared spectroscopy is not yet been effectively used due to inadequate studies to utilize legacy spectral libraries under the field conditions. The performance of such systems is significantly affected by spectral discrepancies created by sample intactness and library differences. In this study, four objectives were devised to obtain directives to address these issues. The first objective was to calibrate and evaluate VNIR models statistically and computationally (i.e. computing resource requirement), using four modeling techniques namely: Partial least squares regression (PLS), Artificial neural networks (ANN), Random forests (RF) and Support vector regression (SVR), to predict soil carbon and nitrogen contents for the Rapid Carbon Assessment (RaCA) project. The second objective was to investigate whether VNIR modeling accuracy can be improved by sample stratification. The third objective was to evaluate the usefulness of these calibrated models to predict external soil samples. The final objective was devised to compare four calibration transfer techniques: Direct Standardization (DS), Piecewise Direct Standardization (PDS), External Parameter Orthogonalization (EPO) and spiking, to transfer field sample scans to laboratory scans of dry ground samples. Results showed that non-linear modeling techniques (ANN, RF and SVR) significantly outperform linear modeling technique (PLS) for all soil properties investigated (accuracy of PLS \u3c RF \u3c SVR ≤ ANN). Local models developed using the four auxiliary variables (Region, land use/land cover class, master horizon and textural class) improved the prediction for all properties (especially for PLS models) compared to the global models (in terms of Root Mean Squared Error of Prediction) with master horizon models outperforming other local models. From the calibration transfer study, it was evident that all the calibration transfer techniques (except for DS) can correct for spectral influences caused by sample intactness. EPO and spiking coupled with ANN model calibration showed the highest performance in accounting for the intactness of samples. These findings will be helpful for future efforts in linking legacy spectra to field spectra for successful implementation of the VNIR sensor systems for vertical or horizontal soil characterization. Advisor Yufeng G

Machine learning based on-line prediction of soil organic carbon after removal of soil moisture effect

It is well-documented in the visible and near-infrared reflectance spectroscopy (VNIRS) studies that soil moisture content (SMC) negatively affects the prediction accuracy of soil attributes. This work was undertaken to remove the negative effect of SMC on the on-line prediction of soil organic carbon (SOC). A mobile VNIR spectrophotometer with a spectral range of 305-1700 nm and spectral resolution of 1 nm (CompactSpec, Tec5 Technology, Germany) was used for the spectral measurements at four farms in Flanders, Belgium. A total of 381 fresh soil samples were collected and divided into a calibration set (264) and a validation set (117). The validation samples were processed (air-dried and grind) and scanned with the same spectrophotometer in the laboratory. Three SMC correction methods, namely, external parameter orthogonalization (EPO), piecewise direct standardization (PDS), and orthogonal signal correction (OSC) were used to correct the on-line fresh spectra based-on its corresponding laboratory spectra. Then, the Cubist machine learning method was used to develop calibration models of SOC using the on-line spectra (after correction) of the calibration set. Results indicated that the EPO-Cubist outperformed the PDS-Cubist and the OSC-Cubist, with considerable improvements in the prediction results of SOC (coefficient of determination (R-2) = 0.76, ratio of performance to deviation (RPD) = 2.08, and root mean square error of prediction (RMSEP) = 0.12%), compared with the corresponding uncorrected on-line spectra (R-2 = 0.55, RPD = 1.24, and RMSEP = 0.20%). It can be concluded that SOC can be accurately predicted on-line using the Cubist machine learning method, after removing the negative effect of SMC with the EPO method

Hyper Spectral Analysis of Soil Iron Oxide using PLSR Method: A Review

Spectroscopy is a rapid, simple, non-destructive and analytical technique, which provides a good alternative that may be used to replace conventional methods of soil analysis. Soil iron oxides occur in almost all type�s soils and they re?ect different environmental conditions by the high variability of their mineralogy and concentration. Soil iron oxide, being an important pedogenic indicator of the soil, measurement of Iron Oxide content can be used as an index of soil fertility. Analytical Spectral Device (ASD) Field Spec 4 Spectroradiometer is used which has 350-2500 nm spectral wavelength range to estimate iron oxide content from the soil sample. The Vis-NIR reflectance spectroscopy requires less effort and it is quick innovation to predict the soil iron oxide content. For collecting the soil iron oxide content from spectral data we are utilizing PLSR which is statistical regression method. This paper states the work that is done on different soil types at different places to observe the iron oxide content in soil

Mapping landscape function with hyperspectral remote sensing of natural grasslands on gold mines

Thesis submitted in fulfilment of the requirements for the degree of Doctor of Philosophy. School of Animal, Plant and Environmental Science, University of the Witwatersrand, Johannesburg, South Africa. October 2016.Mining has negative impacts on the environment in many different ways. One method developed to quantify some of these impacts is Landscape Function Analysis (LFA) and this has been accepted by some mining companies and regulators. In brief, LFA aims at quantifying the organization of vegetative and landscape components in a landscape into patches along a transect and quantifying, in a relative manner, three basic processes important to landscape functioning, namely: soil stability or susceptibility to erosion, infiltration or runoff, and nutrient cycling or organic matter decomposition. However, LFA is limited in large heterogeneous environments, such as those around mining operations, due to its localized nature, and the man hours required to collect a representative set of measurements for such large and complex environments. Remote sensing using satellite-acquired data can overcome these limitations by sampling the entire environment in a rapid and objective manner. What is required is a method of connecting these satellite-based measurements to LFA measurements and then being able to extrapolate these measurements across the entire mine surface. The aim of this research was to develop a method to use satellite-based hyperspectral imagery to predict landscape function analysis (LFA) using partial least squares regression (PLSR). This was broken down into three objectives: (1) Collection of the LFA data in the field and validation of the LFA indices against other environmental variables collected at the same time, (2) validation of PLSR models predicting LFA indices and various environmental variables from ground-based spectra, and (3) production of risk maps based on predicting LFA indices and above-ground biomass using PLSR models and Hyperion satellite-based hyperspectral imagery. Although the study was based in grasslands at two mining regions, West Wits and Vaal River, a suitable Hyperion image was only available for Vaal River. A minimum of 374 points were sampled for LFA indices, ground-based spectra, above-ground biomass and soil cores along 2880 m of LFA transect from both mine sites. Soil cores were weighed fresh before sieving with a 2 mm sieve to separate root and stone fractions. The sieved soil fraction was tested for pH, EC, SOM, and for the West Wits samples, organic nitrogen and total extractable inorganic nitrogen. There was one modification to the LFA method where grass patches were collapsed into homogenous units as it was deemed not feasible to sample 180 m transects at grass tuft scales of 10 – 30 cm, but other patch definitions followed the LFA manual (Tongway and Hindley, 2004). Evidence suggested that some of the different patch types, in particular the bare/biological soil crust – bare grass – sparse grass patch types, represented successional stages in a continuum although this was not conclusive. There also was evidence that the presence or absence of cattle play a role in some processes active in these grasslands and erosion is mainly through deflation, rain splash and sheet wash. Generally the environmental variables supported the LFA indices although the nutrient cycling index was representative of above-ground nutrient cycling but not below-ground nutrient cycling. Models derived with PLSR to predict the LFA indices from ground-based spectral measurements were strong at both mine sites (West Wits: LFA stability r2 = 0.63, P < 0.0001; LFA infiltration r2 = 0.75, P < 0.0001; LFA nutrient cycling r2 = 0.73, P < 0.0001; Vaal River: LFA stability r2 = 0.39, P < 0.0001, LFA infiltration r2 = 0.72, P < 0.0001, LFA nutrient cycling r2 = 0.54, P < 0.0001), as were PLSR models predicting above-ground biomass (West Wits above-ground biomass r2 = 0.55, P = 0.0003; Vaal River above-ground biomass r2 = 0.79, P < 0.0001) and soil moisture (West Wits soil moisture r2 = 0.45, P = 0.0017; Vaal River soil moisture r2 = 0.68, P < 0.0001). However, for soil organic matter (r2 = 0.50, P < 0.0001) and EC (r2 = 0.63, P < 0.0001), Vaal River had strong prediction models while West Wits had weak models for these variables (r2 = 0.31, P = 0.019 and r2 = 0.10 and P < 0.18, respectively). For EC, the wide range of soil values at Vaal River in association with gypsum crusts, and low values throughout West Wits explained these model results but for soil organic matter, no clear explanation for these site differences was identified. Patch-based models could accurately discriminate between spectrally well-defined patch types such S. plumosum patches but were less successful with patch types that were spectrally similar such as the bare/biological soil crust – bare grass – sparse grass patch continuum. Clustering similar patch types together before PLSR modelling did improve these patch-based spectral models. To test the method proposed to predict LFA indices from satellite-based hyperspectral imagery, a Hyperion image matching 6 transects at Vaal River was acquired by NASA’s EO-1 satellite and downloaded from the USGS Glovis website. LFA transects were partitioned to match and extract pixel spectra from the Hyperion data cube. Thirty-one spectra were separated into calibration (20) and validation (11) data. PLSR models were derived from the calibration data, tested with validation data to select the optimum model, and then applied to the entire Hyperion data cube to produce prediction maps for five LFA indices and above-ground biomass. The patch area index (PAI) produced particularly strong models (r2 = 0.79, P = 0.0003, n =11) with validation data, whereas the landscape organization index (LOI) produced weak models. It is argued that this difference between these two essentially similar indices is related to the fact that the PAI is a 2-dimensional index and the LOI is a 1-dimensional index. This difference in these two indices allowed the PAI to compensate for some burned pixels on the transects by “seeing” the density pattern of grass tufts and patches whereas the linear nature of the LOI was more susceptible to the changing dimensions of patch structure due to the effects of fire. Although validation models for the three LFA indices of soil stability, infiltration and nutrient cycling were strong (r2 = 0.72, P = 0.004; r2 = 0.66, P = 0.008; r2 = 0.70, P = 0.005, n = 9 respectively), prediction maps were confounded by the presence of fire on some transects. The poor quality of the Hyperion imagery also meant great care had to be taken in the selection of models to avoid poor quality prediction maps. The 31 bands from the VNIR (478 – 885 nm) portion of the Hyperion spectra were generally the best for PLSR modelling and prediction maps, presumably because of better signal-to-noise ratios due to higher energy in the shorter wavelengths. With two satellite-based hyperspectral sensors already operational, namely the US Hyperion and the Chinese HJ-1A HSI, and a number expected to be launched by various space agencies in the next few years, this research presents a method to use the strengths of LFA and hyperspectral imagery to model and predict LFA index values and thereby produce risk maps of large, heterogeneous landscapes such as mining environments. As this research documents a method of partitioning the landscape rather than the pixel spectra into pure endmembers, it makes a valuable contribution to the fields of landscape ecology and hyperspectral remote sensing.LG201

Proximal hyperspectral imaging detects diurnal and drought-induced changes in maize physiology

Hyperspectral imaging is a promising tool for non-destructive phenotyping of plant physiological traits, which has been transferred from remote to proximal sensing applications, and from manual laboratory setups to automated plant phenotyping platforms. Due to the higher resolution in proximal sensing, illumination variation and plant geometry result in increased non-biological variation in plant spectra that may mask subtle biological differences. Here, a better understanding of spectral measurements for proximal sensing and their application to study drought, developmental and diurnal responses was acquired in a drought case study of maize grown in a greenhouse phenotyping platform with a hyperspectral imaging setup. The use of brightness classification to reduce the illumination-induced non-biological variation is demonstrated, and allowed the detection of diurnal, developmental and early drought-induced changes in maize reflectance and physiology. Diurnal changes in transpiration rate and vapor pressure deficit were significantly correlated with red and red-edge reflectance. Drought-induced changes in effective quantum yield and water potential were accurately predicted using partial least squares regression and the newly developed Water Potential Index 2, respectively. The prediction accuracy of hyperspectral indices and partial least squares regression were similar, as long as a strong relationship between the physiological trait and reflectance was present. This demonstrates that current hyperspectral processing approaches can be used in automated plant phenotyping platforms to monitor physiological traits with a high temporal resolution