Quantitative Mapping of Soil Property Based on Laboratory and Airborne Hyperspectral Data Using Machine Learning

Soil visible and near-infrared spectroscopy provides a non-destructive, rapid and low-cost approach to quantify various soil physical and chemical properties based on their reflectance in the spectral range of 400–2500 nm. With an increasing number of large-scale soil spectral libraries established across the world and new space-borne hyperspectral sensors, there is a need to explore methods to extract informative features from reflectance spectra and produce accurate soil spectroscopic models using machine learning. Features generated from regional or large-scale soil spectral data play a key role in the quantitative spectroscopic model for soil properties. The Land Use/Land Cover Area Frame Survey (LUCAS) soil library was used to explore PLS-derived components and fractal features generated from soil spectra in this study. The gradient-boosting method performed well when coupled with extracted features on the estimation of several soil properties. Transfer learning based on convolutional neural networks (CNNs) was proposed to make the model developed from laboratory data transferable for airborne hyperspectral data. The soil clay map was successfully derived using HyMap imagery and the fine-tuned CNN model developed from LUCAS mineral soils, as deep learning has the potential to learn transferable features that generalise from the source domain to target domain. The external environmental factors like the presence of vegetation restrain the application of imaging spectroscopy. The reflectance data can be transformed into a vegetation suppressed domain with a force invariance approach, the performance of which was evaluated in an agricultural area using CASI airborne hyperspectral data. However, the relationship between vegetation and acquired spectra is complicated, and more efforts should put on removing the effects of external factors to make the model transferable from one sensor to another.:Abstract I Kurzfassung III Table of Contents V List of Figures IX List of Tables XIII List of Abbreviations XV 1 Introduction 1 1.1 Motivation 1 1.2 Soil spectra from different platforms 2 1.3 Soil property quantification using spectral data 4 1.4 Feature representation of soil spectra 5 1.5 Objectives 6 1.6 Thesis structure 7 2 Combining Partial Least Squares and the Gradient-Boosting Method for Soil Property Retrieval Using Visible Near-Infrared Shortwave Infrared Spectra 9 2.1 Abstract 10 2.2 Introduction 10 2.3 Materials and methods 13 2.3.1 The LUCAS soil spectral library 13 2.3.2 Partial least squares algorithm 15 2.3.3 Gradient-Boosted Decision Trees 15 2.3.4 Calculation of relative variable importance 16 2.3.5 Assessment 17 2.4 Results 17 2.4.1 Overview of the spectral measurement 17 2.4.2 Results of PLS regression for the estimation of soil properties 19 2.4.3 Results of PLS-GBDT for the estimation of soil properties 21 2.4.4 Relative important variables derived from PLS regression and the gradient-boosting method 24 2.5 Discussion 28 2.5.1 Dimension reduction for high-dimensional soil spectra 28 2.5.2 GBDT for quantitative soil spectroscopic modelling 29 2.6 Conclusions 30 3 Quantitative Retrieval of Organic Soil Properties from Visible Near-Infrared Shortwave Infrared Spectroscopy Using Fractal-Based Feature Extraction 31 3.1 Abstract 32 3.2 Introduction 32 3.3 Materials and Methods 35 3.3.1 The LUCAS topsoil dataset 35 3.3.2 Fractal feature extraction method 37 3.3.3 Gradient-boosting regression model 37 3.3.4 Evaluation 41 3.4 Results 42 3.4.1 Fractal features for soil spectroscopy 42 3.4.2 Effects of different step and window size on extracted fractal features 45 3.4.3 Modelling soil properties with fractal features 47 3.4.3 Comparison with PLS regression 49 3.5 Discussion 51 3.5.1 The importance of fractal dimension for soil spectra 51 3.5.2 Modelling soil properties with fractal features 52 3.6 Conclusions 53 4 Transfer Learning for Soil Spectroscopy Based on Convolutional Neural Networks and Its Application in Soil Clay Content Mapping Using Hyperspectral Imagery 55 4.1 Abstract 55 4.2 Introduction 56 4.3 Materials and Methods 59 4.3.1 Datasets 59 4.3.2 Methods 62 4.3.3 Assessment 67 4.4 Results and Discussion 67 4.4.1 Interpretation of mineral and organic soils from LUCAS dataset 67 4.4.2 1D-CNN and spectral index for LUCAS soil clay content estimation 69 4.4.3 Application of transfer learning for soil clay content mapping using the pre-trained 1D-CNN model 72 4.4.4 Comparison between spectral index and transfer learning 74 4.4.5 Large-scale soil spectral library for digital soil mapping at the local scale using hyperspectral imagery 75 4.5 Conclusions 75 5 A Case Study of Forced Invariance Approach for Soil Salinity Estimation in Vegetation-Covered Terrain Using Airborne Hyperspectral Imagery 77 5.1 Abstract 78 5.2 Introduction 78 5.3 Materials and Methods 81 5.3.1 Study area of Zhangye Oasis 81 5.3.2 Data description 82 5.3.3 Methods 83 5.3.3 Model performance assessment 85 5.4 Results and Discussion 86 5.4.1 The correlation between NDVI and soil salinity 86 5.4.2 Vegetation suppression performance using the Forced Invariance Approach 86 5.4.3 Estimation of soil properties using airborne hyperspectral data 88 5.5 Conclusions 90 6 Conclusions and Outlook 93 Bibliography 97 Acknowledgements 11

Machine learning based prediction of soil total nitrogen, organic carbon and moisture content by using VIS-NIR spectroscopy

It is widely known that the visible and near infrared (VIS-NIR) spectroscopy has the potential of estimating soil total nitrogen (TN), organic carbon (OC) and moisture content (MC) due to the direct spectral responses these properties have in the near infrared (NIR) region. However, improving the prediction accuracy requires advanced modelling techniques, particularly when measurement is planned for fresh (wet and un-processed) soil samples. The aim of this work is to compare the predictive performance of two linear multivariate and two machine learning methods for TN, OC and MC. The two multivariate methods investigated included principal component regression (PCR) and partial least squares regression (PLSR), whereas the machine learning methods included least squares support vector machines (LS-SVM), and Cubist. A mobile, fibre type, VIS-NIR spectrophotometer was utilised to collect soil spectra (305–2200 nm) in diffuse reflectance mode from 140 wet soil samples collected from one field in Germany. The results indicate that machine learning methods are capable of tackling non-linear problems in the dataset. LS-SVMs and the Cubist method out-performed the linear multivariate methods for the prediction of all three soil properties studied. LS-SVM provided the best prediction for MC (root mean square error of prediction (RMSEP) = 0.457% and residual prediction deviation (RPD) = 2.24) and OC (RMSEP = 0.062% and RPD = 2.20), whereas the Cubist method provided the best prediction for TN (RMSEP = 0.071 and RPD = 1.96)

Soil spectroscopy with the use of chemometrics, machine learning and pre-processing techniques in soil diagnosis: recent advances - a review

Over the past two decades soil spectroscopy, particularly, in the infrared range, is becoming a powerful technique to simplify analysis relative to the traditional chemical methods. It is known as a rapid, cost-effective, quantitative and eco-friendly technique, which can provide hyperspectral data with narrow and numerous wavebands, both in the laboratory and in the field. In this context, the present article reviews the recent developments in mid and near infrared techniques coupled with chemometrics and machine learning tools in addition to the preprocessing transformations and variable selection strategies to diagnose soil physical and chemical properties. Both spectral techniques demonstrated a good ability to provide accurate predictions of specific properties. Moreover, the MIR spectroscopy outperformed NIR for the estimation of most indicators used for fertilizers recommendation. Herein, a detailed overview on the opportunities and challenges that soil spectroscopy offers as efficient diagnostic tool in soil science was provided

Site-specific seeding using multi-sensor and data fusion techniques : a review

Site-specific seeding (SSS) is a precision agricultural (PA) practice aiming at optimizing seeding rate and depth, depending on the within field variability in soil fertility and yield potential. Unlike other site-specific applications, SSS was not adopted sufficiently by farmers due to some technological and practical challenges that need to be overcome. Success of site-specific application strongly depends on the accuracy of measurement of key parameters in the system, modeling and delineation of management zone maps, accurate recommendations and finally the right choice of variable rate (VR) technologies and their integrations. The current study reviews available principles and technologies for both map-based and senor-based SSS. It covers the background of crop and soil quality indicators (SQI), various soil and crop sensor technologies and recommendation approaches of map-based and sensor-based SSS applications. It also discusses the potential of socio-economic benefits of SSS against uniform seeding. The current review proposes prospective future technology synthesis for implementation of SSS in practice. A multi-sensor data fusion system, integrating proper sensor combinations, is suggested as an essential approach for putting SSS into practice