Quantitative Mapping of Soil Property Based on Laboratory and Airborne Hyperspectral Data Using Machine Learning

Soil visible and near-infrared spectroscopy provides a non-destructive, rapid and low-cost approach to quantify various soil physical and chemical properties based on their reflectance in the spectral range of 400–2500 nm. With an increasing number of large-scale soil spectral libraries established across the world and new space-borne hyperspectral sensors, there is a need to explore methods to extract informative features from reflectance spectra and produce accurate soil spectroscopic models using machine learning. Features generated from regional or large-scale soil spectral data play a key role in the quantitative spectroscopic model for soil properties. The Land Use/Land Cover Area Frame Survey (LUCAS) soil library was used to explore PLS-derived components and fractal features generated from soil spectra in this study. The gradient-boosting method performed well when coupled with extracted features on the estimation of several soil properties. Transfer learning based on convolutional neural networks (CNNs) was proposed to make the model developed from laboratory data transferable for airborne hyperspectral data. The soil clay map was successfully derived using HyMap imagery and the fine-tuned CNN model developed from LUCAS mineral soils, as deep learning has the potential to learn transferable features that generalise from the source domain to target domain. The external environmental factors like the presence of vegetation restrain the application of imaging spectroscopy. The reflectance data can be transformed into a vegetation suppressed domain with a force invariance approach, the performance of which was evaluated in an agricultural area using CASI airborne hyperspectral data. However, the relationship between vegetation and acquired spectra is complicated, and more efforts should put on removing the effects of external factors to make the model transferable from one sensor to another.:Abstract I Kurzfassung III Table of Contents V List of Figures IX List of Tables XIII List of Abbreviations XV 1 Introduction 1 1.1 Motivation 1 1.2 Soil spectra from different platforms 2 1.3 Soil property quantification using spectral data 4 1.4 Feature representation of soil spectra 5 1.5 Objectives 6 1.6 Thesis structure 7 2 Combining Partial Least Squares and the Gradient-Boosting Method for Soil Property Retrieval Using Visible Near-Infrared Shortwave Infrared Spectra 9 2.1 Abstract 10 2.2 Introduction 10 2.3 Materials and methods 13 2.3.1 The LUCAS soil spectral library 13 2.3.2 Partial least squares algorithm 15 2.3.3 Gradient-Boosted Decision Trees 15 2.3.4 Calculation of relative variable importance 16 2.3.5 Assessment 17 2.4 Results 17 2.4.1 Overview of the spectral measurement 17 2.4.2 Results of PLS regression for the estimation of soil properties 19 2.4.3 Results of PLS-GBDT for the estimation of soil properties 21 2.4.4 Relative important variables derived from PLS regression and the gradient-boosting method 24 2.5 Discussion 28 2.5.1 Dimension reduction for high-dimensional soil spectra 28 2.5.2 GBDT for quantitative soil spectroscopic modelling 29 2.6 Conclusions 30 3 Quantitative Retrieval of Organic Soil Properties from Visible Near-Infrared Shortwave Infrared Spectroscopy Using Fractal-Based Feature Extraction 31 3.1 Abstract 32 3.2 Introduction 32 3.3 Materials and Methods 35 3.3.1 The LUCAS topsoil dataset 35 3.3.2 Fractal feature extraction method 37 3.3.3 Gradient-boosting regression model 37 3.3.4 Evaluation 41 3.4 Results 42 3.4.1 Fractal features for soil spectroscopy 42 3.4.2 Effects of different step and window size on extracted fractal features 45 3.4.3 Modelling soil properties with fractal features 47 3.4.3 Comparison with PLS regression 49 3.5 Discussion 51 3.5.1 The importance of fractal dimension for soil spectra 51 3.5.2 Modelling soil properties with fractal features 52 3.6 Conclusions 53 4 Transfer Learning for Soil Spectroscopy Based on Convolutional Neural Networks and Its Application in Soil Clay Content Mapping Using Hyperspectral Imagery 55 4.1 Abstract 55 4.2 Introduction 56 4.3 Materials and Methods 59 4.3.1 Datasets 59 4.3.2 Methods 62 4.3.3 Assessment 67 4.4 Results and Discussion 67 4.4.1 Interpretation of mineral and organic soils from LUCAS dataset 67 4.4.2 1D-CNN and spectral index for LUCAS soil clay content estimation 69 4.4.3 Application of transfer learning for soil clay content mapping using the pre-trained 1D-CNN model 72 4.4.4 Comparison between spectral index and transfer learning 74 4.4.5 Large-scale soil spectral library for digital soil mapping at the local scale using hyperspectral imagery 75 4.5 Conclusions 75 5 A Case Study of Forced Invariance Approach for Soil Salinity Estimation in Vegetation-Covered Terrain Using Airborne Hyperspectral Imagery 77 5.1 Abstract 78 5.2 Introduction 78 5.3 Materials and Methods 81 5.3.1 Study area of Zhangye Oasis 81 5.3.2 Data description 82 5.3.3 Methods 83 5.3.3 Model performance assessment 85 5.4 Results and Discussion 86 5.4.1 The correlation between NDVI and soil salinity 86 5.4.2 Vegetation suppression performance using the Forced Invariance Approach 86 5.4.3 Estimation of soil properties using airborne hyperspectral data 88 5.5 Conclusions 90 6 Conclusions and Outlook 93 Bibliography 97 Acknowledgements 11

Visible and near infrared spectroscopy in soil science

This chapter provides a review on the state of soil visible–near infrared (vis–NIR) spectroscopy. Our intention is for the review to serve as a source of up-to date information on the past and current role of vis–NIR spectroscopy in soil science. It should also provide critical discussion on issues surrounding the use of vis–NIR for soil analysis and on future directions. To this end, we describe the fundamentals of visible and infrared diffuse reflectance spectroscopy and spectroscopic multivariate calibrations. A review of the past and current role of vis–NIR spectroscopy in soil analysis is provided, focusing on important soil attributes such as soil organic matter (SOM), minerals, texture, nutrients, water, pH, and heavy metals. We then discuss the performance and generalization capacity of vis–NIR calibrations, with particular attention on sample pre-tratments, co-variations in data sets, and mathematical data preprocessing. Field analyses and strategies for the practical use of vis–NIR are considered. We conclude that the technique is useful to measure soil water and mineral composition and to derive robust calibrations for SOM and clay content. Many studies show that we also can predict properties such as pH and nutrients, although their robustness may be questioned. For future work we recommend that research should focus on: (i) moving forward with more theoretical calibrations, (ii) better understanding of the complexity of soil and the physical basis for soil reflection, and (iii) applications and the use of spectra for soil mapping and monitoring, and for making inferences about soils quality, fertility and function. To do this, research in soil spectroscopy needs to be more collaborative and strategic. The development of the Global Soil Spectral Library might be a step in the right direction

Diffuse reflectance spectroscopy for estimation of soil organic matter and texture in burned and abandoned cropland areas

<br /

How to measure, report and verify soil carbon change to realize the potential of soil carbon sequestration for atmospheric greenhouse gas removal

Global Change Biology published by John Wiley & Sons Ltd There is growing international interest in better managing soils to increase soil organic carbon (SOC) content to contribute to climate change mitigation, to enhance resilience to climate change and to underpin food security, through initiatives such as international ‘4p1000’ initiative and the FAO\u27s Global assessment of SOC sequestration potential (GSOCseq) programme. Since SOC content of soils cannot be easily measured, a key barrier to implementing programmes to increase SOC at large scale, is the need for credible and reliable measurement/monitoring, reporting and verification (MRV) platforms, both for national reporting and for emissions trading. Without such platforms, investments could be considered risky. In this paper, we review methods and challenges of measuring SOC change directly in soils, before examining some recent novel developments that show promise for quantifying SOC. We describe how repeat soil surveys are used to estimate changes in SOC over time, and how long-term experiments and space-for-time substitution sites can serve as sources of knowledge and can be used to test models, and as potential benchmark sites in global frameworks to estimate SOC change. We briefly consider models that can be used to simulate and project change in SOC and examine the MRV platforms for SOC change already in use in various countries/regions. In the final section, we bring together the various components described in this review, to describe a new vision for a global framework for MRV of SOC change, to support national and international initiatives seeking to effect change in the way we manage our soils

Remote Sensing for Precision Agriculture: Sentinel-2 Improved Features and Applications

The use of satellites to monitor crops and support their management is gathering increasing attention. The improved temporal, spatial, and spectral resolution of the European Space Agency (ESA) launched Sentinel-2 A + B twin platform is paving the way to their popularization in precision agriculture. Besides the Sentinel-2 A + B constellation technical features the open-access nature of the information they generate, and the available support software are a significant improvement for agricultural monitoring. This paper was motivated by the challenges faced by researchers and agrarian institutions entering this field; it aims to frame remote sensing principles and Sentinel-2 applications in agriculture. Thus, we reviewed the features and uses of Sentinel-2 in precision agriculture, including abiotic and biotic stress detection, and agricultural management. We also compared the panoply of satellites currently in use for land remote sensing that are relevant for agriculture to the Sentinel-2 A + B constellation features. Contrasted with previous satellite image systems, the Sentinel-2 A + B twin platform has dramatically increased the capabilities for agricultural monitoring and crop management worldwide. Regarding crop stress monitoring, Sentinel-2 capacities for abiotic and biotic stresses detection represent a great step forward in many ways though not without its limitations; therefore, combinations of field data and different remote sensing techniques may still be needed. We conclude that Sentinel-2 has a wide range of useful applications in agriculture, yet still with room for further improvements. Current and future ways that Sentinel-2 can be utilized are also discussed.This research was funded by the Spanish projects AGL2016-76527-R and IRUEC PCIN-2017-063 from the Ministerio de Economía y Competividad (MINECO, Spain) and by the support of Catalan Institution for Research and Advanced Studies (ICREA, Generalitat de Catalunya, Spain), through the ICREA Academia Program

Finding an efficient method to measure soil carbon pools

Primarily, this thesis addresses the need for a cost and time efficient methodology to quantify both chemically and physically stabilized soil organic carbon (SOC) fractions. The time and cost saving would be gained if the need to physically and chemically separate soil samples to measure carbons of different fractions could be reduced. Some progress has been made by quantifying SOC using spectroscopy techniques (NIR and MIR) in place of combustion, which both give quantitative and qualitative (organic functional groups) measurement for SOC. What remains is to see if the comprehensive data that can be gained from the spectroscopic techniques can be used to quantify and characterize SOC in soil fractions while using minimal or no physical and/or fractionation pretreatments. Chapter 2 of this thesis is a comprehensive review of the literature focusing on soil C pools and significant aggregate theories currently developed. From the literature, it is apparent that soil aggregation is mediated by SOC and aggregate fractions of various sizes are stabilized by a range of SOC functional groups. At the same time SOC and aggregation models were being developed SOC pools were also being defined to populate soil carbon turnover models (e.g. Roth-C). Only recently, has research been undertaken which attempt to combine soil aggregation with soil C turnover models. A limitation to using these models is the time taken to separate aggregate and carbon fractions and the review of the literature indicates that using spectroscopic techniques may be a solution to this

Finding an efficient method to measure soil carbon pools

Primarily, this thesis addresses the need for a cost and time efficient methodology to quantify both chemically and physically stabilized soil organic carbon (SOC) fractions. The time and cost saving would be gained if the need to physically and chemically separate soil samples to measure carbons of different fractions could be reduced. Some progress has been made by quantifying SOC using spectroscopy techniques (NIR and MIR) in place of combustion, which both give quantitative and qualitative (organic functional groups) measurement for SOC. What remains is to see if the comprehensive data that can be gained from the spectroscopic techniques can be used to quantify and characterize SOC in soil fractions while using minimal or no physical and/or fractionation pretreatments. Chapter 2 of this thesis is a comprehensive review of the literature focusing on soil C pools and significant aggregate theories currently developed. From the literature, it is apparent that soil aggregation is mediated by SOC and aggregate fractions of various sizes are stabilized by a range of SOC functional groups. At the same time SOC and aggregation models were being developed SOC pools were also being defined to populate soil carbon turnover models (e.g. Roth-C). Only recently, has research been undertaken which attempt to combine soil aggregation with soil C turnover models. A limitation to using these models is the time taken to separate aggregate and carbon fractions and the review of the literature indicates that using spectroscopic techniques may be a solution to this

Remote Sensing for Precision Agriculture: Sentinel-2 Improved Features and Applications

The use of satellites to monitor crops and support their management is gathering increasing attention. The improved temporal, spatial, and spectral resolution of the European Space Agency (ESA) launched Sentinel-2 A + B twin platform is paving the way to their popularization in precision agriculture. Besides the Sentinel-2 A + B constellation technical features the open-access nature of the information they generate, and the available support software are a significant improvement for agricultural monitoring. This paper was motivated by the challenges faced by researchers and agrarian institutions entering this field; it aims to frame remote sensing principles and Sentinel-2 applications in agriculture. Thus, we reviewed the features and uses of Sentinel-2 in precision agriculture, including abiotic and biotic stress detection, and agricultural management. We also compared the panoply of satellites currently in use for land remote sensing that are relevant for agriculture to the Sentinel-2 A + B constellation features. Contrasted with previous satellite image systems, the Sentinel-2 A + B twin platform has dramatically increased the capabilities for agricultural monitoring and crop management worldwide. Regarding crop stress monitoring, Sentinel-2 capacities for abiotic and biotic stresses detection represent a great step forward in many ways though not without its limitations; therefore, combinations of field data and different remote sensing techniques may still be needed. We conclude that Sentinel-2 has a wide range of useful applications in agriculture, yet still with room for further improvements. Current and future ways that Sentinel-2 can be utilized are also discusse

Predicting soil organic carbon in a small farm system using in situ spectral measurements and the random forest regression

A research report submitted to the Faculty of Science, University of the Witwatersrand, Johannesburg, in partial fulfillment of the requirements for the degree of Master of Science (Geographical Information Sciences and Remote Sensing) Johannesburg, 2017Soil organic carbon is considered as the most determining indicator of soil fertility. The purpose of this research was to predict the soil organic carbon in the Mokhotlong region, eastern of Lesotho using in situ spectral measurements and random forest regression. Soil reflectance spectra were acquired by a portable field spectrometer. The performance of random forest regression was assessed by comparing it with one of the most popular models in spectroscopy, partial least square regression. Laboratory spectroscopy measurements of the soil samples were analysed for assessing the accuracy of in situ spectroscopy based-models. The effect of the Savitzky−Golay first derivative in improving partial least square regression and random forest regression in both spectral data was also assessed. The results indicated that the random forest regression could accurately predict the soil organic carbon contents on an independent dataset using in situ spectroscopy data (RPD = 3.77, Rp2= 0.88, RMSEP = 0.64%). The overall best predictive model was achieved with the derivative laboratory spectral data using random forest with the optimum number of key wavelengths (RPD = 3.77, Rp2= 0.88, RMSEP = 0.64%). In contrast, partial least square regression was likely to overfit the calibration dataset. Important wavelengths to predict soil organic contents were localised around the visible range (400-700 nm). An implication of this research is that soil organic carbon can accurately be estimated using derivative in situ spectroscopy measurements and random forest regression with key wavelengths.MT 201

The application of proximal visible and near-infrared spectroscopy to estimate soil organic matter on the Triffa Plain of Morocco

Soil organic matter (SOM) is a fundamental soil constituent. The estimation of this parameter in the laboratory using the classical method is complex time-consuming and requires the use of chemical reagents. The objectives of this study were to assess the accuracy of two laboratory measurement setups of the VIS-NIR spectroscopy in estimating SOM content and determine the important spectral bands in the SOM estimation model. A total of 115 soil samples were collected from the non-root zone (0-20 cm) of soil in the study area of the Triffa Plain and then analysed for SOM in the laboratory by the Walkley-Black method. The reflectance spectra of soil samples were measured by two protocols, Contact Probe (CP) and Pistol Grip (PG)) of the ASD spectroradiometer (350-2500 nm) in the laboratory. Partial least squares regression (PLSR) was used to develop the prediction models. The results of coefficient of determination (R-2) and the root mean square error (RMSE) showed that the pistol grip offers reasonable accuracy with an R-2 = 0.93 and RMSE = 0.13 compared to the contact probe protocol with an R-2 = 0.85 and RMSE = 0.19. The near-Infrared range were more accurate than those in the visible range for predicting SOM using the both setups (CP and PG). The significant wavelengths contributing to the prediction of SOM for (PG) setup were at: 424, 597, 1432, 1484, 1830,1920, 2200, 2357 and 2430 nm, while were at 433, 587, 1380, 1431, 1929, 2200 and 2345 nm for (CP) setup