Remote sensing image fusion on 3D scenarios: A review of applications for agriculture and forestry

Three-dimensional (3D) image mapping of real-world scenarios has a great potential to provide the user with a more accurate scene understanding. This will enable, among others, unsupervised automatic sampling of meaningful material classes from the target area for adaptive semi-supervised deep learning techniques. This path is already being taken by the recent and fast-developing research in computational fields, however, some issues related to computationally expensive processes in the integration of multi-source sensing data remain. Recent studies focused on Earth observation and characterization are enhanced by the proliferation of Unmanned Aerial Vehicles (UAV) and sensors able to capture massive datasets with a high spatial resolution. In this scope, many approaches have been presented for 3D modeling, remote sensing, image processing and mapping, and multi-source data fusion. This survey aims to present a summary of previous work according to the most relevant contributions for the reconstruction and analysis of 3D models of real scenarios using multispectral, thermal and hyperspectral imagery. Surveyed applications are focused on agriculture and forestry since these fields concentrate most applications and are widely studied. Many challenges are currently being overcome by recent methods based on the reconstruction of multi-sensorial 3D scenarios. In parallel, the processing of large image datasets has recently been accelerated by General-Purpose Graphics Processing Unit (GPGPU) approaches that are also summarized in this work. Finally, as a conclusion, some open issues and future research directions are presented.European Commission 1381202-GEU PYC20-RE-005-UJA IEG-2021Junta de Andalucia 1381202-GEU PYC20-RE-005-UJA IEG-2021Instituto de Estudios GiennesesEuropean CommissionSpanish Government UIDB/04033/2020DATI-Digital Agriculture TechnologiesPortuguese Foundation for Science and Technology 1381202-GEU FPU19/0010

Hyperspectral Imaging from Ground Based Mobile Platforms and Applications in Precision Agriculture

This thesis focuses on the use of line scanning hyperspectral sensors on mobile ground based platforms and applying them to agricultural applications. First this work deals with the geometric and radiometric calibration and correction of acquired hyperspectral data. When operating at low altitudes, changing lighting conditions are common and inevitable, complicating the retrieval of a surface's reflectance, which is solely a function of its physical structure and chemical composition. Therefore, this thesis contributes the evaluation of an approach to compensate for changes in illumination and obtain reflectance that is less labour intensive than traditional empirical methods. Convenient field protocols are produced that only require a representative set of illumination and reflectance spectral samples. In addition, a method for determining a line scanning camera's rigid 6 degree of freedom (DOF) offset and uncertainty with respect to a navigation system is developed, enabling accurate georegistration and sensor fusion. The thesis then applies the data captured from the platform to two different agricultural applications. The first is a self-supervised weed detection framework that allows training of a per-pixel classifier using hyperspectral data without manual labelling. The experiments support the effectiveness of the framework, rivalling classifiers trained on hand labelled training data. Then the thesis demonstrates the mapping of mango maturity using hyperspectral data on an orchard wide scale using efficient image scanning techniques, which is a world first result. A novel classification, regression and mapping pipeline is proposed to generate per tree mango maturity averages. The results confirm that maturity prediction in mango orchards is possible in natural daylight using a hyperspectral camera, despite complex micro-illumination-climates under the canopy

Hyperspectral Unmixing Overview: Geometrical, Statistical, and Sparse Regression-Based Approaches

Imaging spectrometers measure electromagnetic energy scattered in their instantaneous field view in hundreds or thousands of spectral channels with higher spectral resolution than multispectral cameras. Imaging spectrometers are therefore often referred to as hyperspectral cameras (HSCs). Higher spectral resolution enables material identification via spectroscopic analysis, which facilitates countless applications that require identifying materials in scenarios unsuitable for classical spectroscopic analysis. Due to low spatial resolution of HSCs, microscopic material mixing, and multiple scattering, spectra measured by HSCs are mixtures of spectra of materials in a scene. Thus, accurate estimation requires unmixing. Pixels are assumed to be mixtures of a few materials, called endmembers. Unmixing involves estimating all or some of: the number of endmembers, their spectral signatures, and their abundances at each pixel. Unmixing is a challenging, ill-posed inverse problem because of model inaccuracies, observation noise, environmental conditions, endmember variability, and data set size. Researchers have devised and investigated many models searching for robust, stable, tractable, and accurate unmixing algorithms. This paper presents an overview of unmixing methods from the time of Keshava and Mustard's unmixing tutorial [1] to the present. Mixing models are first discussed. Signal-subspace, geometrical, statistical, sparsity-based, and spatial-contextual unmixing algorithms are described. Mathematical problems and potential solutions are described. Algorithm characteristics are illustrated experimentally.Comment: This work has been accepted for publication in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensin

Multispectral image analysis in laparoscopy – A machine learning approach to live perfusion monitoring

Modern visceral surgery is often performed through small incisions. Compared to open surgery, these minimally invasive interventions result in smaller scars, fewer complications and a quicker recovery. While to the patients benefit, it has the drawback of limiting the physician’s perception largely to that of visual feedback through a camera mounted on a rod lens: the laparoscope. Conventional laparoscopes are limited by “imitating” the human eye. Multispectral cameras remove this arbitrary restriction of recording only red, green and blue colors. Instead, they capture many specific bands of light. Although these could help characterize important indications such as ischemia and early stage adenoma, the lack of powerful digital image processing prevents realizing the technique’s full potential. The primary objective of this thesis was to pioneer fluent functional multispectral imaging (MSI) in laparoscopy. The main technical obstacles were: (1) The lack of image analysis concepts that provide both high accuracy and speed. (2) Multispectral image recording is slow, typically ranging from seconds to minutes. (3) Obtaining a quantitative ground truth for the measurements is hard or even impossible. To overcome these hurdles and enable functional laparoscopy, for the first time in this field physical models are combined with powerful machine learning techniques. The physical model is employed to create highly accurate simulations, which in turn teach the algorithm to rapidly relate multispectral pixels to underlying functional changes. To reduce the domain shift introduced by learning from simulations, a novel transfer learning approach automatically adapts generic simulations to match almost arbitrary recordings of visceral tissue. In combination with the only available video-rate capable multispectral sensor, the method pioneers fluent perfusion monitoring with MSI. This system was carefully tested in a multistage process, involving in silico quantitative evaluations, tissue phantoms and a porcine study. Clinical applicability was ensured through in-patient recordings in the context of partial nephrectomy; in these, the novel system characterized ischemia live during the intervention. Verified against a fluorescence reference, the results indicate that fluent, non-invasive ischemia detection and monitoring is now possible. In conclusion, this thesis presents the first multispectral laparoscope capable of videorate functional analysis. The system was successfully evaluated in in-patient trials, and future work should be directed towards evaluation of the system in a larger study. Due to the broad applicability and the large potential clinical benefit of the presented functional estimation approach, I am confident the descendants of this system are an integral part of the next generation OR

Computational Spectral Imaging: A Contemporary Overview

Spectral imaging collects and processes information along spatial and spectral coordinates quantified in discrete voxels, which can be treated as a 3D spectral data cube. The spectral images (SIs) allow identifying objects, crops, and materials in the scene through their spectral behavior. Since most spectral optical systems can only employ 1D or maximum 2D sensors, it is challenging to directly acquire the 3D information from available commercial sensors. As an alternative, computational spectral imaging (CSI) has emerged as a sensing tool where the 3D data can be obtained using 2D encoded projections. Then, a computational recovery process must be employed to retrieve the SI. CSI enables the development of snapshot optical systems that reduce acquisition time and provide low computational storage costs compared to conventional scanning systems. Recent advances in deep learning (DL) have allowed the design of data-driven CSI to improve the SI reconstruction or, even more, perform high-level tasks such as classification, unmixing, or anomaly detection directly from 2D encoded projections. This work summarises the advances in CSI, starting with SI and its relevance; continuing with the most relevant compressive spectral optical systems. Then, CSI with DL will be introduced, and the recent advances in combining the physical optical design with computational DL algorithms to solve high-level tasks

Crop Disease Detection Using Remote Sensing Image Analysis

Pest and crop disease threats are often estimated by complex changes in crops and the applied agricultural practices that result mainly from the increasing food demand and climate change at global level. In an attempt to explore high-end and sustainable solutions for both pest and crop disease management, remote sensing technologies have been employed, taking advantages of possible changes deriving from relative alterations in the metabolic activity of infected crops which in turn are highly associated to crop spectral reflectance properties. Recent developments applied to high resolution data acquired with remote sensing tools, offer an additional tool which is the opportunity of mapping the infected field areas in the form of patchy land areas or those areas that are susceptible to diseases. This makes easier the discrimination between healthy and diseased crops, providing an additional tool to crop monitoring. The current book brings together recent research work comprising of innovative applications that involve novel remote sensing approaches and their applications oriented to crop disease detection. The book provides an in-depth view of the developments in remote sensing and explores its potential to assess health status in crops

Mehitamata õhusõiduki rakendamine põllukultuuride saagikuse ja maa harimisviiside tuvastamisel

A Thesis for applying for the degree of Doctor of Philosophy in Environmental Protection.Väitekiri filosoofiadoktori kraadi taotlemiseks keskkonnakaitse erialal.This thesis aims to examine how machine learning (ML) technologies have aided significant advancements in image analysis in the area of precision agriculture. These multimodal computing technologies extend the use of machine learning to a broader spectrum of data collecting and selection for the advancement of agricultural practices (Nawar et al., 2017) These techniques will assist complicated cropping systems with more informed decisions with less human intervention, and provide a scalable framework for incorporating expert knowledge of the PA system. (Chlingaryan et al., 2018). Complexity, on the other hand, can be seen as a disadvantage in crop trials, as machine learning models require training/testing databases, limited areas with insignificant sampling sizes, time and space-specificity, and environmental factor interventions, all of which complicate parameter selection and make using a single empirical model for an entire region impractical. During the early stages of writing this thesis, we used a relatively traditional machine learning method to address the regression problem of crop yield and biomass prediction [(i.e., random forest regression (RFR), support vector regression (SVR), and artificial neural network (ANN)] to predicted dry matter (DM) yields of red clover. It obtained favourable results, however, the choosing of hyperparameters, the lengthy algorithms selection process, data cleaning, and redundant collinearity issues significantly limited the way of the machine learning application. We will further discuss the recent trend of automated machine learning (AutoML) that has been driving further significant technological innovation in the application of artificial intelligence from its automated algorithm selection and hyperparameter optimization of the deployable pipeline model for unravelling substance problems. However, a present knowledge gap exists in the integration of machine learning (ML) technology with unmanned aerial systems (UAS) and hyperspectral-based imaging data categorization and regression applications. In this thesis, we explored a state-of-the-art (SOTA) and entirely open-source AutoML framework, Auto-sklearn, which was built on one of the most frequently used machine learning systems, Scikit-learn. It was integrated with two unique AutoML visualization tools to examine the recognition and acceptance of multispectral vegetation indices (VI) data collected from UAS and hyperspectral narrow-band VIs across a varied spectrum of agricultural management practices (AMP). These procedures incorporate soil tillage method (STM), cultivation method (CM), and manure application (MA), and are classified as four-crop combination fields (i.e., red clover-grass mixture, spring wheat, pea-oat mixture, and spring barley). Additionally, they have not been thoroughly evaluated and lack characteristics that are accessible in agriculture remote sensing applications. This thesis further explores the existing gaps in the knowledge base for several critical crop categories and cultivation management methods referring to biomass and yield analysis, as well as to gain a better understanding of the potential for remotely sensed solutions to field-based and multifunctional platforms to meet precision agriculture demands. To overcome these knowledge gaps, this research introduces a rapid, non-destructive, and low-cost framework for field-based biomass and grain yield modelling, as well as the identification of agricultural management practices. The results may aid agronomists and farmers in establishing more accurate agricultural methods and in monitoring environmental conditions more effectively.Doktoritöö eesmärk oli uurida, kuidas masinõppe (MÕ) tehnoloogiad võimaldavad edusamme täppispõllumajanduse valdkonna pildianalüüsis. Multimodaalsed arvutustehnoloogiad laiendavad masinõppe kasutamist põllumajanduses andmete kogumisel ja valimisel (Nawar et al., 2017). Selline täpsemal informatsioonil põhinev tehnoloogia võimaldab keerukate viljelussüsteemide puhul teha otsuseid inimese vähema sekkumisega, ja loob skaleeritava raamistiku täppispõllumajanduse jaoks (Chlingaryan et al., 2018). Põllukultuuride katsete korral on komplekssete masinõppemudelite kasutamine keerukas, sest alad on piiratud ning valimi suurus ei ole piisav; vaja on testandmebaase, kindlaid aja- ja ruumitingimusi ning keskkonnategureid. See komplitseerib parameetrite valikut ning muudab ebapraktiliseks ühe empiirilise mudeli kasutamise terves piirkonnas. Siinse uurimuse algetapis rakendati suhteliselt traditsioonilist masinõppemeetodit, et lahendada saagikuse ja biomassi prognoosimise regressiooniprobleem (otsustusmetsa regression, tugivektori regressioon ja tehisnärvivõrk) punase ristiku prognoositava kuivaine saagikuse suhtes. Saadi sobivaid tulemusi, kuid hüperparameetrite valimine, pikk algoritmide valimisprotsess, andmete puhastamine ja kollineaarsusprobleemid takistasid masinõpet oluliselt. Automatiseeritud masinõppe (AMÕ) uusimate suundumustena rakendatakse tehisintellekti, et lahendada põhiprobleemid automatiseeritud algoritmi valiku ja rakendatava pipeline-mudeli hüperparameetrite optimeerimise abil. Seni napib teadmisi MÕ tehnoloogia integreerimiseks mehitamata õhusõidukite ning hüperspektripõhiste pildiandmete kategoriseerimise ja regressioonirakendustega. Väitekirjas uuriti nüüdisaegset ja avatud lähtekoodiga AMÕ tehnoloogiat Auto-sklearn, mis on ühe enimkasutatava masinõppesüsteemi Scikit-learn edasiarendus. Süsteemiga liideti kaks unikaalset AMÕ visualiseerimisrakendust, et uurida mehitamata õhusõidukiga kogutud andmete multispektraalsete taimkatteindeksite ja hüperspektraalsete kitsaribaandmete taimkatteindeksite tuvastamist ja rakendamist põllumajanduses. Neid võtteid kasutatakse mullaharimisel, kultiveerimisel ja sõnnikuga väetamisel nelja kultuuriga põldudel (punase ristiku rohusegu, suvinisu, herne-kaera segu, suvioder). Neid ei ole põhjalikult hinnatud, samuti ei hõlma need omadusi, mida kasutatatakse põllumajanduses kaugseire rakendustes. Uurimus käsitleb biomassi ja saagikuse seni uurimata analüüsivõimalusi oluliste põllukultuuride ja viljelusmeetodite näitel. Hinnatakse ka kaugseirelahenduste potentsiaali põllupõhiste ja multifunktsionaalsete platvormide kasutamisel täppispõllumajanduses. Uurimus tutvustab kiiret, keskkonna suhtes kahjutut ja mõõduka hinnaga tehnoloogiat põllupõhise biomassi ja teraviljasaagi modelleerimiseks, et leida sobiv viljelusviis. Töö tulemused võimaldavad põllumajandustootjatel ja agronoomidel tõhusamalt valida põllundustehnoloogiaid ning arvestada täpsemalt keskkonnatingimustega.Publication of this thesis is supported by the Estonian University of Life Scieces and by the Doctoral School of Earth Sciences and Ecology created under the auspices of the European Social Fund

Hyperspectral Imaging for Fine to Medium Scale Applications in Environmental Sciences

The aim of the Special Issue “Hyperspectral Imaging for Fine to Medium Scale Applications in Environmental Sciences” was to present a selection of innovative studies using hyperspectral imaging (HSI) in different thematic fields. This intention reflects the technical developments in the last three decades, which have brought the capacity of HSI to provide spectrally, spatially and temporally detailed data, favoured by e.g., hyperspectral snapshot technologies, miniaturized hyperspectral sensors and hyperspectral microscopy imaging. The present book comprises a suite of papers in various fields of environmental sciences—geology/mineral exploration, digital soil mapping, mapping and characterization of vegetation, and sensing of water bodies (including under-ice and underwater applications). In addition, there are two rather methodically/technically-oriented contributions dealing with the optimized processing of UAV data and on the design and test of a multi-channel optical receiver for ground-based applications. All in all, this compilation documents that HSI is a multi-faceted research topic and will remain so in the future

A Broadly Tunable Surface Plasmon-Coupled Wavelength Filter for Visible and Near Infrared Hyperspectral Imaging

Hyperspectral imaging is a set of techniques that has contributed to the study of advanced materials, pharmaceuticals, semiconductors, ceramics, polymers, biological specimens, and geological samples. Its use for remote sensing has advanced our understanding of agriculture, forestry, the Earth, environmental science, and the universe. The development of ultra-compact handheld hyperspectral imagers has been impeded by the scarcity of small widefield tunable wavelength filters. The widefield modality is preferred for handheld imaging applications in which image registration can be performed to counter scene shift caused by irregular user motions that would thwart scanning approaches. In the work presented here an electronically tunable widefield wavelength filter has been developed for hyperspectral imaging applications in the visible and near-infrared region. Conventional electronically tunable widefield imaging filter technologies include liquid crystal-based filters, acousto-optic tunable filters, and electronically tuned etalons; each having its own set of advantages and disadvantages. The construction of tunable filters is often complex and requires elaborate optical assemblies and electronic control circuits. I introduce in the work presented here is a novel widefield tunable filter, the surface plasmon coupled tunable filter (SPCTF), for visible and near infrared imaging. The SPCTF is based on surface plasmon coupling and has simple optical design that can be miniaturized without sacrificing performance. The SPCTF provides diffraction limited spatial resolution with a moderately narrow nominal passband (\u3c10 \u3enm) and a large spurious free spectral range (450 nm-1000 nm). The SPCTF employs surface plasmon coupling of the π-polarized component of incident light in metal films separated by a tunable dielectric layer. Acting on the π-polarized component, the device is limited to transmitting 50 percent of unpolarized incident light. This is higher than the throughput of comparable Lyot-based liquid crystal tunable filters that employ a series of linear polarizers. In addition, the SPCTF is not susceptible to the unwanted harmonic bands that lead to spurious diffraction in Bragg-based devices. Hence its spurious free spectral range covers a broad region from the blue through near infrared wavelengths. The compact design and rugged optical assembly make it suitable for hand-held hyperspectral imagers. The underlying theory and SPCTF design are presented along with a comparison of its performance to calculated estimates of transmittance, spectral resolution, and spectral range. In addition, widefield hyperspectral imaging using the SPCTF is demonstrated on model sample