Multisource and Multitemporal Data Fusion in Remote Sensing

The sharp and recent increase in the availability of data captured by different sensors combined with their considerably heterogeneous natures poses a serious challenge for the effective and efficient processing of remotely sensed data. Such an increase in remote sensing and ancillary datasets, however, opens up the possibility of utilizing multimodal datasets in a joint manner to further improve the performance of the processing approaches with respect to the application at hand. Multisource data fusion has, therefore, received enormous attention from researchers worldwide for a wide variety of applications. Moreover, thanks to the revisit capability of several spaceborne sensors, the integration of the temporal information with the spatial and/or spectral/backscattering information of the remotely sensed data is possible and helps to move from a representation of 2D/3D data to 4D data structures, where the time variable adds new information as well as challenges for the information extraction algorithms. There are a huge number of research works dedicated to multisource and multitemporal data fusion, but the methods for the fusion of different modalities have expanded in different paths according to each research community. This paper brings together the advances of multisource and multitemporal data fusion approaches with respect to different research communities and provides a thorough and discipline-specific starting point for researchers at different levels (i.e., students, researchers, and senior researchers) willing to conduct novel investigations on this challenging topic by supplying sufficient detail and references

Cybergis-enabled remote sensing data analytics for deep learning of landscape patterns and dynamics

Mapping landscape patterns and dynamics is essential to various scientific domains and many practical applications. The availability of large-scale and high-resolution light detection and ranging (LiDAR) remote sensing data provides tremendous opportunities to unveil complex landscape patterns and better understand landscape dynamics from a 3D perspective. LiDAR data have been applied to diverse remote sensing applications where large-scale landscape mapping is among the most important topics. While researchers have used LiDAR for understanding landscape patterns and dynamics in many fields, to fully reap the benefits and potential of LiDAR is increasingly dependent on advanced cyberGIS and deep learning approaches. In this context, the central goal of this dissertation is to develop a suite of innovative cyberGIS-enabled deep-learning frameworks for combining LiDAR and optical remote sensing data to analyze landscape patterns and dynamics with four interrelated studies. The first study demonstrates a high-accuracy land-cover mapping method by integrating 3D information from LiDAR with multi-temporal remote sensing data using a 3D deep-learning model. The second study combines a point-based classification algorithm and an object-oriented change detection strategy for urban building change detection using deep learning. The third study develops a deep learning model for accurate hydrological streamline detection using LiDAR, which has paved a new way of harnessing LiDAR data to map landscape patterns and dynamics at unprecedented computational and spatiotemporal scales. The fourth study resolves computational challenges in handling remote sensing big data and deep learning of landscape feature extraction and classification through a cutting-edge cyberGIS approach

Monitoring Cloud-prone Complex Landscapes At Multiple Spatial Scales Using Medium And High Resolution Optical Data: A Case Study In Central Africa

Tracking land surface dynamics over cloud-prone areas with complex mountainous terrain and a landscape that is heterogeneous at a scale of approximately 10 m, is an important challenge in the remote sensing of tropical regions in developing nations, due to the small plot sizes. Persistent monitoring of natural resources in these regions at multiple spatial scales requires development of tools to identify emerging land cover transformation due to anthropogenic causes, such as agricultural expansion and climate change. Along with the cloud cover and obstructions by topographic distortions due to steep terrain, there are limitations to the accuracy of monitoring change using available historical satellite imagery, largely due to sparse data access and the lack of high quality ground truth for classifier training. One such complex region is the Lake Kivu region in Central Africa. This work addressed these problems to create an effective process for monitoring the Lake Kivu region located in Central Africa. The Lake Kivu region is a biodiversity hotspot with a complex and heterogeneous landscape and intensive agricultural development, where individual plot sizes are often at the scale of 10m. Procedures were developed that use optical data from satellite and aerial observations at multiple scales to tackle the monitoring challenges. First, a novel processing chain was developed to systematically monitor the spatio-temporal land cover dynamics of this region over the years 1988, 2001, and 2011 using Landsat data, complemented by ancillary data. Topographic compensation was performed on Landsat reflectances to avoid the strong illumination angle impacts and image compositing was used to compensate for frequent cloud cover and thus incomplete annual data availability in the archive. A systematic supervised classification, using the state-of-the-art machine learning classifier Random Forest, was applied to the composite Landsat imagery to obtain land cover thematic maps with overall accuracies of 90% and higher. Subsequent change analysis between these years found extensive conversions of the natural environment as a result of human related activities. The gross forest cover loss for 1988-2001 and 2001- 2011 periods was 216.4 and 130.5 thousand hectares, respectively, signifying significant deforestation in the period of civil war and a relatively stable and lower deforestation rate later, possibly due to conservation and reforestation efforts in the region. The other dominant land cover changes in the region were aggressive subsistence farming and urban expansion displacing natural vegetation and arable lands. Despite limited data availability, this study fills the gap of much needed detailed and updated land cover change information for this biologically important region of Central Africa. While useful on a regional scale, Landsat data can be inadequate for more detailed studies of land cover change. Based on an increasing availability of high resolution imagery and light detection and ranging (LiDAR) data from manned and unmanned aerial platforms (\u3c1m \u3eresolution), a study was performed leading to a novel generic framework for land cover monitoring at fine spatial scales. The approach fuses high spatial resolution aerial imagery and LiDAR data to produce land cover maps with high spatial detail using object-based image analysis techniques. The classification framework was tested for a scene with both natural and cultural features and was found to be more than 90 percent accurate, sufficient for detailed land cover change studies

Multitemporal Feature-Level Fusion on Hyperspectral and LiDAR Data in the Urban Environment

publishedVersio

Mapping a European spruce bark beetle outbreak using sentinel-2 remote sensing data

Insect outbreaks affect forests, causing the deaths of trees and high economic loss. In this study, we explored the detection of European spruce bark beetle (Ips typographus, L.) outbreaks at the individual tree crown level using multispectral satellite images. Moreover, we explored the possibility of tracking the progression of the outbreak over time using multitemporal data. Sentinel-2 data acquired during the summer of 2020 over a bark beetle–infested area in the Italian Alps were used for the mapping and tracking over time, while airborne lidar data were used to automatically detect the individual tree crowns and to classify tree species. Mapping and tracking of the outbreak were carried out using a support vector machine classifier with input vegetation indices extracted from the multispectral data. The results showed that it was possible to detect two stages of the outbreak (i.e., early, and late) with an overall accuracy of 83.4%. Moreover, we showed how it is technically possible to track the evolution of the outbreak in an almost bi-weekly period at the level of the individual tree crowns. The outcomes of this paper are useful from both a management and ecological perspective: it allows forest managers to map a bark beetle outbreak at different stages with a high spatial accuracy, and the maps describing the evolution of the outbreak could be used in further studies related to the behavior of bark beetle

Challenges and Opportunities of Multimodality and Data Fusion in Remote Sensing

International audience—Remote sensing is one of the most common ways to extract relevant information about the Earth and our environment. Remote sensing acquisitions can be done by both active (synthetic aperture radar, LiDAR) and passive (optical and thermal range, multispectral and hyperspectral) devices. According to the sensor, a variety of information about the Earth's surface can be obtained. The data acquired by these sensors can provide information about the structure (optical, synthetic aperture radar), elevation (LiDAR) and material content (multi and hyperspectral) of the objects in the image. Once considered together their comple-mentarity can be helpful for characterizing land use (urban analysis, precision agriculture), damage detection (e.g., in natural disasters such as floods, hurricanes, earthquakes, oil-spills in seas), and give insights to potential exploitation of resources (oil fields, minerals). In addition, repeated acquisitions of a scene at different times allows one to monitor natural resources and environmental variables (vegetation phenology, snow cover), anthropological effects (urban sprawl, deforestation), climate changes (desertification, coastal erosion) among others. In this paper, we sketch the current opportunities and challenges related to the exploitation of multimodal data for Earth observation. This is done by leveraging the outcomes of the Data Fusion contests, organized by the IEEE Geoscience and Remote Sensing Society since 2006. We will report on the outcomes of these contests, presenting the multimodal sets of data made available to the community each year, the targeted applications and an analysis of the submitted methods and results: How was multimodality considered and integrated in the processing chain? What were the improvements/new opportunities offered by the fusion? What were the objectives to be addressed and the reported solutions? And from this, what will be the next challenges

New Computational Methods for Automated Large-Scale Archaeological Site Detection

Aquesta tesi doctoral presenta una sèrie d'enfocaments, fluxos de treball i models innovadors en el camp de l'arqueologia computacional per a la detecció automatitzada a gran escala de jaciments arqueològics. S'introdueixen nous conceptes, enfocaments i estratègies, com ara lidar multitemporal, aprenentatge automàtic híbrid, refinament, curriculum learning i blob analysis; així com diferents mètodes d'augment de dades aplicats per primera vegada en el camp de l'arqueologia. S'utilitzen múltiples fonts, com ara imatges de satèl·lits multiespectrals, fotografies RGB de plataformes VANT, mapes històrics i diverses combinacions de sensors, dades i fonts. Els mètodes creats durant el desenvolupament d'aquest doctorat s'han avaluat en projectes en curs: Urbanització a Hispània i la Gàl·lia Mediterrània en el primer mil·lenni aC, detecció de monticles funeraris utilitzant algorismes d'aprenentatge automàtic al nord-oest de la Península Ibèrica, prospecció arqueològica intel·ligent basada en drons (DIASur), i cartografiat del patrimoni arqueològic al sud d'Àsia (MAHSA), per a la qual s'han dissenyat fluxos de treball adaptats als reptes específics del projecte. Aquests nous mètodes han aconseguit proporcionar solucions als problemes comuns d'estudis arqueològics presents en estudis similars, com la baixa precisió en detecció i les poques dades d'entrenament. Els mètodes validats i presentats com a part de la tesi doctoral s'han publicat en accés obert amb el codi disponible perquè puguin implementar-se en altres estudis arqueològics.Esta tesis doctoral presenta una serie de enfoques, flujos de trabajo y modelos innovadores en el campo de la arqueología computacional para la detección automatizada a gran escala de yacimientos arqueológicos. Se introducen nuevos conceptos, enfoques y estrategias, como lidar multitemporal, aprendizaje automático híbrido, refinamiento, curriculum learning y blob analysis; así como diferentes métodos de aumento de datos aplicados por primera vez en el campo de la arqueología. Se utilizan múltiples fuentes, como lidar, imágenes satelitales multiespectrales, fotografías RGB de plataformas VANT, mapas históricos y varias combinaciones de sensores, datos y fuentes. Los métodos creados durante el desarrollo de este doctorado han sido evaluados en proyectos en curso: Urbanización en Iberia y la Galia Mediterránea en el Primer Milenio a. C., Detección de túmulos mediante algoritmos de aprendizaje automático en el Noroeste de la Península Ibérica, Prospección Arqueológica Inteligente basada en Drones (DIASur), y cartografiado del Patrimonio del Sur de Asia (MAHSA), para los que se han diseñado flujos de trabajo adaptados a los retos específicos del proyecto. Estos nuevos métodos han logrado proporcionar soluciones a problemas comunes de la prospección arqueológica presentes en estudios similares, como la baja precisión en detección y los pocos datos de entrenamiento. Los métodos validados y presentados como parte de la tesis doctoral se han publicado en acceso abierto con su código disponible para que puedan implementarse en otros estudios arqueológicos.This doctoral thesis presents a series of innovative approaches, workflows and models in the field of computational archaeology for the automated large-scale detection of archaeological sites. New concepts, approaches and strategies are introduced such as multitemporal lidar, hybrid machine learning, refinement, curriculum learning and blob analysis; as well as different data augmentation methods applied for the first time in the field of archaeology. Multiple sources are used, such as lidar, multispectral satellite imagery, RGB photographs from UAV platform, historical maps, and several combinations of sensors, data, and sources. The methods created during the development of this PhD have been evaluated in ongoing projects: Urbanization in Iberia and Mediterranean Gaul in the First Millennium BC, Detection of burial mounds using machine learning algorithms in the Northwest of the Iberian Peninsula, Drone-based Intelligent Archaeological Survey (DIASur), and Mapping Archaeological Heritage in South Asia (MAHSA), for which workflows adapted to the project’ s specific challenges have been designed. These new methods have managed to provide solutions to common archaeological survey problems, presented in similar large-scale site detection studies, such as the low precision in previous detection studies and how to handle problems with few training data. The validated approaches for site detection presented as part of the PhD have been published as open access papers with freely available code so can be implemented in other archaeological studies

Investigating the potential for detecting Oak Decline using Unmanned Aerial Vehicle (UAV) Remote Sensing

This PhD project develops methods for the assessment of forest condition utilising modern remote sensing technologies, in particular optical imagery from unmanned aerial systems and with Structure from Motion photogrammetry. The research focuses on health threats to the UK’s native oak trees, specifically, Chronic Oak Decline (COD) and Acute Oak Decline (AOD). The data requirements and methods to identify these complex diseases are investigatedusing RGB and multispectral imagery with very high spatial resolution, as well as crown textural information. These image data are produced photogrammetrically from multitemporal unmanned aerial vehicle (UAV) flights, collected during different seasons to assess the influence of phenology on the ability to detect oak decline. Particular attention is given to the identification of declined oak health within the context of semi-natural forests and heterogenous stands. Semi-natural forest environments pose challenges regarding naturally occurring variability. The studies investigate the potential and practical implications of UAV remote sensing approaches for detection of oak decline under these conditions. COD is studied at Speculation Cannop, a section in the Forest of Dean, dominated by 200-year-old oaks, where decline symptoms have been present for the last decade. Monks Wood, a semi-natural woodland in Cambridgeshire, is the study site for AOD, where trees exhibit active decline symptoms. Field surveys at these sites are designed and carried out to produce highly-accurate differential GNSS positional information of symptomatic and control oak trees. This allows the UAV data to be related to COD or AOD symptoms and the validation of model predictions. Random Forest modelling is used to determine the explanatory value of remote sensing-derived metrics to distinguish trees affected by COD or AOD from control trees. Spectral and textural variables are extracted from the remote sensing data using an object-based approach, adopting circular plots around crown centres at individual tree level. Furthermore, acquired UAV imagery is applied to generate a species distribution map, improving on the number of detectable species and spatial resolution from a previous classification using multispectral data from a piloted aircraft. In the production of the map, parameters relevant for classification accuracy, and identification of oak in particular, are assessed. The effect of plot size, sample size and data combinations are studied. With optimised parameters for species classification, the updated species map is subsequently employed to perform a wall-to-wall prediction of individual oak tree condition, evaluating the potential of a full inventory detection of declined health. UAV-acquired data showed potential for discrimination of control trees and declined trees, in the case of COD and AOD. The greatest potential for detecting declined oak condition was demonstrated with narrowband multispectral imagery. Broadband RGB imagery was determined to be unsuitable for a robust distinction between declined and control trees. The greatest explanatory power was found in remotely-sensed spectra related to photosynthetic activity, indicated by the high feature importance of nearinfrared spectra and the vegetation indices NDRE and NDVI. High feature importance was also produced by texture metrics, that describe structural variations within the crown. The findings indicate that the remotely sensed explanatory variables hold significant information regarding changes in leaf chemistry and crown morphology that relate to chlorosis, defoliation and dieback occurring in the course of the decline. In the case of COD, a distinction of symptomatic from control trees was achieved with 75 % accuracy. Models developed for AOD detection yielded AUC scores up to 0.98,when validated on independent sample data. Classification of oak presence was achieved with a User’s accuracy of 97 % and the produced species map generated 95 % overall accuracy across the eight species within the study area in the north-east of Monks Wood. Despite these encouraging results, it was shown that the generalisation of models is unfeasible at this stage and many challenges remain. A wall-to-wall prediction of decline status confirmed the inability to generalise, yielding unrealistic results, with a high number of declined trees predicted. Identified weaknesses of the developed models indicate complexity related to the natural variability of heterogenous forests combined with the diverse symptoms of oak decline. Specific to the presented studies, additional limitations were attributed to limited ground truth, consequent overfitting,the binary classification of oak health status and uncertainty in UAV-acquired reflectance values. Suggestions for future work are given and involve the extension of field sampling with a non-binary dependent variable to reflect the severity of oak decline induced stress. Further technical research on the quality and reliability of UAV remote sensing data is also required

Automatic differentiation of Eucalyptus species through Sentinel-2 images, Worldview-3 images and LiDAR data

Eucalyptus constitutes one of the most common tree genera used in forest plantations worldwide. In Europe, Eucalyptus trees are especially common in the Northwest of the Iberian Peninsula, E. nitens and E. globulus being the most commonly cultivated species. Each species presents particularities that lend to different exploitation strategies and industrial usages. Therefore, updated knowledge about the abundance and spatial distribution of the different species is important for forest planning. This is a special challenge for areas where forest land is highly fragmented. Remote sensing has been used to efficiently monitor the distribution of the Eucalyptus genera, however little research has been able to map specific Eucalyptus species. This study evaluates the efficiency of Sentinel-2 data, Worldview-3 images, and Airborne LiDAR data in the differentiation of E. nitens and E. globulus. Supervised classifications were performed using neural networks for these data sets both individually and in combination. The highest accuracies were obtained when using Sentinel-2 data in combination with LiDAR point clouds and when using Sentinel-2 data in a multitemporal approach. The best time of year to differentiate between the two species is during the emergence of spring shoots. Worldview-3 images have a moderate capacity to differentiate between the two species, although this is increased when textural metrics are included. This study can serve as the basis for generating Eucalyptus species distribution maps, which will allow for improved forest management and planning.Xunta de GaliciaAgencia Estatal de Investigación | Ref. PID2019-111581RB-I00Universidade de Vigo/CISU