Advances in Object and Activity Detection in Remote Sensing Imagery

The recent revolution in deep learning has enabled considerable development in the fields of object and activity detection. Visual object detection tries to find objects of target classes with precise localisation in an image and assign each object instance a corresponding class label. At the same time, activity recognition aims to determine the actions or activities of an agent or group of agents based on sensor or video observation data. It is a very important and challenging problem to detect, identify, track, and understand the behaviour of objects through images and videos taken by various cameras. Together, objects and their activity recognition in imaging data captured by remote sensing platforms is a highly dynamic and challenging research topic. During the last decade, there has been significant growth in the number of publications in the field of object and activity recognition. In particular, many researchers have proposed application domains to identify objects and their specific behaviours from air and spaceborne imagery. This Special Issue includes papers that explore novel and challenging topics for object and activity detection in remote sensing images and videos acquired by diverse platforms

Contribution to ground-based and UAV SAR systems for Earth observation

Mankind's way of life is the main driver of a planetary-scale change that is marked by the growing of human population's demand of energy, food, goods, services and information. As a result, it have emerged new ecological, economical, social and geopolitical concerns. In this scenario, SAR remote sensing is a potential tool that provides unique information about the Earth's properties and processes that can be used to solve societal challenges of local and global dimension. SARs, which are coherent systems that are able to provide high resolution images with weather independence, represent a suitable alternative for EO with diverse applications. Some examples of SAR application areas are topography (DEM generation with interferometry), agriculture (crop classification or soil moisture), or geology (monitoring surface deformation). In this framework, the encompassing objective of the present doctoral work has been part of the implementation and the subsequent evaluation of capabilities of two X-band SAR sensors. On the one hand, the RISKSAR-X radar designed to be operated at ground to monitor small-scale areas of observation and, on the other, the ARBRES-X sensor designed to be integrated into small UAVs. Despite its inherently dissimilar conception, the concurrence of both sensors has been evidenced along this manuscript. By taking advantage of the similarities between them, it has been possible to analogously assess both sensors to obtain conclusions. In this context, the common link has been the development of the polarimetric OtF operation mode of the RISKSAR-X, allowing this sensor to be operated equivalently to the ARBRES-X. Regarding the RISKSAR-X SAR sensor, several hardware contributions have been developed during part of this Ph.D. with the aim of improving the system performance. By endowing the system with the capability to operate in the fully polarimetric OtF acquisition mode, the relative long scanning time has been reduced. It is of great interest since the measured scatterers that present a short term variable reflectivity during the scanning time, such as moving vegetation, may degrade the extracted parameters from the retrieved data and the SAR image reconstruction. During this doctoral activity, it has been studied the image blurring, the decorrelation and the coherence degradation introduced by this effect. Furthermore, a new term in the differential interferometric coherence that takes into account the image blurring has been introduced. Concerning the ARBRES-X SAR system, one of the main objectives pursued during this Ph.D. has been the integration of the sensor into a small UAV MP overcoming restrictions of weight, size and aerodynamics of the platform. The use of this type of platforms is expected to open up new possibilities in airborne SAR remote sensing, since it offers much more versatility than the commonly used fixed wings UAVs. Different innovative flight strategies with this type of platforms have been assessed and some preliminary results have been obtained with the use of the ARBRES-X SAR system. During the course of the present doctoral work, much effort has been devoted to achieve the first experimental repeat-pass interfereometric results obtained with the UAV MP together with the ARBRES-X. Moreover, the sensor has been endowed with fully polarimetric capabilities by applying the improvements developed to the RISKSAR-X radar, which is another example of the duality between both systems. Finally, a vertical and a semicircular aperture have been successfully performed obtaining SLC images of the scenario, which envisages the capability of the UAV MP to perform tomographic images and complete circular apertures in the future. In conclusion, the UAV MP is a promising platform that opens new potentials for several applications, such as repeat-pass interferometry or differential tomography imaging with the realization of almost arbitrary trajectories.El mode de viure de la humanitat és el principal motor d'un canvi a escala planetària que està marcat per la creixent demanda d'energia, d'aliment, de béns, de serveis i d'informació de les poblacions humanes. Com a resultat, han sorgit noves inquietuds ecològiques, econòmiques, socials i geopolítiques. En aquest escenari, la detecció remota SAR és una eina potencial que proporciona informació única sobre les propietats i processos de la Terra que es pot utilitzar per resoldre reptes socials de dimensió local i global. Els SARs, que són sistemes coherents que poden proporcionar imatges d'alta resolució amb independència del temps, representen una alternativa adequada per a l'observació de la Terra. Alguns exemples d'àrees d'aplicació SAR són la topografia (generació de DEM amb interferometria), l'agricultura (classificació de cultius o humitat del sòl) o la geologia (monitoratge de deformació superficial). En aquest context, l'objectiu general del present doctorat ha estat part de la implementació i posterior avaluació de les capacitats de dos sensors SAR de banda X. D'una banda, el radar RISKSAR-X dissenyat per funcionar a terra i monitoritzar àrees d'observació a petita escala i, d'altra, el sensor ARBRES-X dissenyat per ser integrat en petits UAVs. Malgrat la seva concepció inherentment diferent, la concurrència d'ambdós sensors s'ha evidenciat al llarg d'aquest manuscrit. Aprofitant les similituds entre ells, s'han pogut avaluar de forma anàloga els dos sensors per obtenir conclusions. En aquest sentit, el vincle comú ha estat el desenvolupament del mode de funcionament polimètric OtF del RISKSAR-X, permetent que aquest sensor operi de forma equivalent a l'ARBRES-X. Pel que fa al sensor RISKSAR-X, s'han desenvolupat diverses contribucions hardware durant part d'aquest doctorat amb l'objectiu de millorar el rendiment del sistema. En dotar el sistema de la possibilitat d'operar en el mode d'adquisició totalment polarimètric OtF, s'ha reduït el relatiu llarg temps d'escaneig. Això és de gran interès ja que els blancs mesurats que presenten una reflectivitat variable a curt termini, com ara la vegetació en moviment, poden degradar els paràmetres extrets de les dades recuperades i la reconstrucció d'imatges SAR. Durant aquesta activitat doctoral s'ha estudiat el desenfocat de la imatge, la decorrelació i la degradació de la coherència introduïts per aquest efecte. A més, s'ha introduït un nou terme en la coherència interferomètrica diferencial que té en compte el desenfocat de la imatge. Pel que fa al sistema ARBRES-X, un dels principals objectius perseguits durant aquest doctorat ha estat la integració del sensor en un petit UAV MP superant les restriccions de pes, grandària i aerodinàmica de la plataforma. S'espera que l'ús d'aquest tipus de plataformes obri noves possibilitats en la detecció remota SAR aerotransportada, ja que ofereix molta més versatilitat que els UAV d'ales fixes habituals. S'han avaluat diferents estratègies de vol innovadores amb aquest tipus de plataformes i s'han obtingut resultats preliminars amb l'ús del sistema ARBRES-X. Durant el transcurs del present treball, s'ha dedicat molt esforç a assolir els primers resultats experimentals d'interferometria de múltiple passada obtinguts amb l'UAV MP conjuntament amb l'ARBRES-X. A més, el sensor ha estat dotat de capacitats totalment polarimètriques aplicant les millores desenvolupades al radar RISKSAR-X, el qual constitueix un altre exemple de la dualitat entre ambdós sistemes. Finalment, s'han realitzat amb èxit una apertura vertical i semicircular obtenint imatges SLC de l'escenari, el qual permet preveure la capacitat de l'UAV MP per a realitzar imatges tomogràfiques i apertures circulars completes en el futur. En conclusió, l'UAV MP és una plataforma prometedora que obre nous potencials per a diverses aplicacions, com ara la interferometria de múltiple passada o la tomografia diferencial amb la realització de trajectòries gairebé arbitràries.Postprint (published version

Contribution to ground-based and UAV SAR systems for Earth observation

Mankind's way of life is the main driver of a planetary-scale change that is marked by the growing of human population's demand of energy, food, goods, services and information. As a result, it have emerged new ecological, economical, social and geopolitical concerns. In this scenario, SAR remote sensing is a potential tool that provides unique information about the Earth's properties and processes that can be used to solve societal challenges of local and global dimension. SARs, which are coherent systems that are able to provide high resolution images with weather independence, represent a suitable alternative for EO with diverse applications. Some examples of SAR application areas are topography (DEM generation with interferometry), agriculture (crop classification or soil moisture), or geology (monitoring surface deformation). In this framework, the encompassing objective of the present doctoral work has been part of the implementation and the subsequent evaluation of capabilities of two X-band SAR sensors. On the one hand, the RISKSAR-X radar designed to be operated at ground to monitor small-scale areas of observation and, on the other, the ARBRES-X sensor designed to be integrated into small UAVs. Despite its inherently dissimilar conception, the concurrence of both sensors has been evidenced along this manuscript. By taking advantage of the similarities between them, it has been possible to analogously assess both sensors to obtain conclusions. In this context, the common link has been the development of the polarimetric OtF operation mode of the RISKSAR-X, allowing this sensor to be operated equivalently to the ARBRES-X. Regarding the RISKSAR-X SAR sensor, several hardware contributions have been developed during part of this Ph.D. with the aim of improving the system performance. By endowing the system with the capability to operate in the fully polarimetric OtF acquisition mode, the relative long scanning time has been reduced. It is of great interest since the measured scatterers that present a short term variable reflectivity during the scanning time, such as moving vegetation, may degrade the extracted parameters from the retrieved data and the SAR image reconstruction. During this doctoral activity, it has been studied the image blurring, the decorrelation and the coherence degradation introduced by this effect. Furthermore, a new term in the differential interferometric coherence that takes into account the image blurring has been introduced. Concerning the ARBRES-X SAR system, one of the main objectives pursued during this Ph.D. has been the integration of the sensor into a small UAV MP overcoming restrictions of weight, size and aerodynamics of the platform. The use of this type of platforms is expected to open up new possibilities in airborne SAR remote sensing, since it offers much more versatility than the commonly used fixed wings UAVs. Different innovative flight strategies with this type of platforms have been assessed and some preliminary results have been obtained with the use of the ARBRES-X SAR system. During the course of the present doctoral work, much effort has been devoted to achieve the first experimental repeat-pass interfereometric results obtained with the UAV MP together with the ARBRES-X. Moreover, the sensor has been endowed with fully polarimetric capabilities by applying the improvements developed to the RISKSAR-X radar, which is another example of the duality between both systems. Finally, a vertical and a semicircular aperture have been successfully performed obtaining SLC images of the scenario, which envisages the capability of the UAV MP to perform tomographic images and complete circular apertures in the future. In conclusion, the UAV MP is a promising platform that opens new potentials for several applications, such as repeat-pass interferometry or differential tomography imaging with the realization of almost arbitrary trajectories.El mode de viure de la humanitat és el principal motor d'un canvi a escala planetària que està marcat per la creixent demanda d'energia, d'aliment, de béns, de serveis i d'informació de les poblacions humanes. Com a resultat, han sorgit noves inquietuds ecològiques, econòmiques, socials i geopolítiques. En aquest escenari, la detecció remota SAR és una eina potencial que proporciona informació única sobre les propietats i processos de la Terra que es pot utilitzar per resoldre reptes socials de dimensió local i global. Els SARs, que són sistemes coherents que poden proporcionar imatges d'alta resolució amb independència del temps, representen una alternativa adequada per a l'observació de la Terra. Alguns exemples d'àrees d'aplicació SAR són la topografia (generació de DEM amb interferometria), l'agricultura (classificació de cultius o humitat del sòl) o la geologia (monitoratge de deformació superficial). En aquest context, l'objectiu general del present doctorat ha estat part de la implementació i posterior avaluació de les capacitats de dos sensors SAR de banda X. D'una banda, el radar RISKSAR-X dissenyat per funcionar a terra i monitoritzar àrees d'observació a petita escala i, d'altra, el sensor ARBRES-X dissenyat per ser integrat en petits UAVs. Malgrat la seva concepció inherentment diferent, la concurrència d'ambdós sensors s'ha evidenciat al llarg d'aquest manuscrit. Aprofitant les similituds entre ells, s'han pogut avaluar de forma anàloga els dos sensors per obtenir conclusions. En aquest sentit, el vincle comú ha estat el desenvolupament del mode de funcionament polimètric OtF del RISKSAR-X, permetent que aquest sensor operi de forma equivalent a l'ARBRES-X. Pel que fa al sensor RISKSAR-X, s'han desenvolupat diverses contribucions hardware durant part d'aquest doctorat amb l'objectiu de millorar el rendiment del sistema. En dotar el sistema de la possibilitat d'operar en el mode d'adquisició totalment polarimètric OtF, s'ha reduït el relatiu llarg temps d'escaneig. Això és de gran interès ja que els blancs mesurats que presenten una reflectivitat variable a curt termini, com ara la vegetació en moviment, poden degradar els paràmetres extrets de les dades recuperades i la reconstrucció d'imatges SAR. Durant aquesta activitat doctoral s'ha estudiat el desenfocat de la imatge, la decorrelació i la degradació de la coherència introduïts per aquest efecte. A més, s'ha introduït un nou terme en la coherència interferomètrica diferencial que té en compte el desenfocat de la imatge. Pel que fa al sistema ARBRES-X, un dels principals objectius perseguits durant aquest doctorat ha estat la integració del sensor en un petit UAV MP superant les restriccions de pes, grandària i aerodinàmica de la plataforma. S'espera que l'ús d'aquest tipus de plataformes obri noves possibilitats en la detecció remota SAR aerotransportada, ja que ofereix molta més versatilitat que els UAV d'ales fixes habituals. S'han avaluat diferents estratègies de vol innovadores amb aquest tipus de plataformes i s'han obtingut resultats preliminars amb l'ús del sistema ARBRES-X. Durant el transcurs del present treball, s'ha dedicat molt esforç a assolir els primers resultats experimentals d'interferometria de múltiple passada obtinguts amb l'UAV MP conjuntament amb l'ARBRES-X. A més, el sensor ha estat dotat de capacitats totalment polarimètriques aplicant les millores desenvolupades al radar RISKSAR-X, el qual constitueix un altre exemple de la dualitat entre ambdós sistemes. Finalment, s'han realitzat amb èxit una apertura vertical i semicircular obtenint imatges SLC de l'escenari, el qual permet preveure la capacitat de l'UAV MP per a realitzar imatges tomogràfiques i apertures circulars completes en el futur. En conclusió, l'UAV MP és una plataforma prometedora que obre nous potencials per a diverses aplicacions, com ara la interferometria de múltiple passada o la tomografia diferencial amb la realització de trajectòries gairebé arbitràries

Generation of a Land Cover Atlas of environmental critic zones using unconventional tools

L'abstract è presente nell'allegato / the abstract is in the attachmen

Mutual superimposing of SAR and ground-level shooting images mediated by intermediate multi-altitude images

When satellite-based SAR (Synthetic Aperture Radar) images and images acquired from the ground are registered, they offer a wealth of information such as topographic, vegetation or water surface to be extracted from the ground-level shooting images. Simultaneously, high temporal-resolution and high spatial-resolution information obtained by the ground-level shooting images can be superimposed on satellite images. However, due to the differences in imaging modality, spatial resolutions, and observation angle, it was not easy to directly extract the corresponding points between them. This paper proposes an image registration method to estimate the correspondence between SAR images and ground-level shooting images through a set of multi-altitude images taken at different heights

Panorama Generation for Stereoscopic Visualization of Large-Scale Scenes

In this thesis, we address the problem of modeling and stereoscopically visualizing large-scale scenes captured with a single moving camera. In many applications that image large-scale scenes the critical information desired is the 3D spatial information of stationary objects and movers within the scene. Stereo panoramas, like regular panoramas, provide a wide field-of-view that can represent the entire scene, with the stereo panoramas additionally representing the motion parallax and allowing for 3D visualization and reconstruction of the scene. The primary issue with stereo panorama construction methods is that they are constrained for a particular camera motion model; typically the camera is constrained to move along a linear or circular path. Here we present a method for constructing stereo panoramas for general camera motion, and we develop a (1) Unified Stereo Mosaic Framework that handles general camera motion models. To construct stereo panoramas for general motion we created a new (2) Stereo Mosaic Layering algorithm that speeds up panorama construction enabling real-time applications. In large-scale scene applications it is often the case that the scene will be imaged persistently by passing over the same path multiple times or two or more sensors of different modalities will pass over the the same scene. To address these issues we developed methods for (3) Multi-Run and Multi-Modal Mosaic Alignment. Finally, we developed an (4) Intelligent Stereo Visualization that allows a viewer to interact and stereoscopically view the stereo panoramas developed from general motion

USING UNMANNED AERIAL SYSTEMS TO ESTIMATE CROWN AREA-BIOMASS RELATIONSHIPS IN A NORWAY SPRUCE (Picea abies) PLANTATION

The main purpose of this study is to evaluate the potential improvements in forest biomass estimation by using commercially accessible and affordable small Unmanned Aerial System (sUAS) imagery. For this goal, total aboveground biomass is quantified by: field enumeration of all n = 1,919 trees in a Norway spruce (Picea abies) plantation; obtaining UAS imagery from over the plantation; indirect estimation of tree size distribution using UAS imagery; comparison of two methods for estimating tree crown area from UAS imagery; and linear correlations between field measurements and photo-based crown area estimates. Results showed generally weak correlation between field measurements of above-ground biomass and UAS-derived crown area estimates. To improve forest biomass estimation, recommended refinements were detailed

Deep Learning Methods for Remote Sensing

Remote sensing is a field where important physical characteristics of an area are exacted using emitted radiation generally captured by satellite cameras, sensors onboard aerial vehicles, etc. Captured data help researchers develop solutions to sense and detect various characteristics such as forest fires, flooding, changes in urban areas, crop diseases, soil moisture, etc. The recent impressive progress in artificial intelligence (AI) and deep learning has sparked innovations in technologies, algorithms, and approaches and led to results that were unachievable until recently in multiple areas, among them remote sensing. This book consists of sixteen peer-reviewed papers covering new advances in the use of AI for remote sensing

Verbesserte Dokumentation des kulturellen Erbes mithilfe digitaler Photogrammetrie mit sichtbaren und thermischen Bildern von unbemannten Luftfahrzeugen (UAV)

There is always need for reliable and accurate data for documentation of cultural heritage including archaeological areas. The development in 3D data acquisition has let some technologies use for getting a complete documentation. Close range photogrammetry and terrestrial laser scanning are among the most common used techniques which help to get 3D data acquisition, with high level of detail, accuracy and effective results. However, these techniques are not always the most suitable ones for large archaeological areas, yet aerial images may help to provide a general overview of the area which is fundamental for interpretation and documentation of archaeological sites. Because of the limitations in aerial photogrammetry, UAVs (Unmanned Aerial Vehicles) has become an optimal solution for archaeological areas documentation with its potentials in the context of costs and abilities. To cover large areas at different altitudes, to be able to fly at different altitudes, under different weather conditions, to acquire image with high resolution are among the main advantages of this technology which make it usable and preferable for archaeological documentation. Since UAVs have been rapidly improving in sophistication and reliability, its possibilities aid in archaeological research have recently generated much interest, particularly for documenting sites, monuments and excavations. In this case study several aerial surveys will be conducted with a UAV mounted thermal camera on an archaeological area. After acquiring aerial images, they will be processed for producing both color and thermal-imagery in related software. Next step will be the alignment of the images in order to build an accurate and georeferenced 3D and mesh model of surveyed area. Then colored and thermal orthophoto mosaics as well as digital surface model (DSM) will be obtained for the documentation. The datasets of thermal images and color images will be collected and compared in order to detect archaeological remains on and under the ground.Es besteht immer Bedarf an zuverlässigen und genauen Daten für die Dokumentation des kulturellen Erbes, einschließlich archäologischer Gebiete. Die technischen Entwicklungen in der 3D-Datenerfassung haben erst die vollständige Dokumentation ermöglicht. Nahbereichsphotogrammetrie und terrestrisches Laserscanning gehören zu den am häufigsten verwendeten Techniken, die 3D-Datenerfassung mit hohem Detaillierungsgrad, Genauigkeit und effektive Ergebnissen ermöglichen. Diese Techniken sind jedoch nicht immer die am besten geeigneten für große archäologische Gebiete, dennoch können Luftbilder helfen, einen allgemeinen Überblick über das Gebiet zu geben, was für die Interpretation und Dokumentation archäologischer Stätten von grundlegender Bedeutung ist. Aufgrund der Einschränkungen in der Luftbildvermessung sind UAVs (Unmanned Aerial Vehicles) zu einer optimalen Lösung für die archäologische Geländedokumentation mit ihren Potenzialen im Kontext von Kosten und Fähigkeiten geworden. Hauptvorteile dieser Technologie sind u.a. große Gebiete in verschiedenen Höhen abzudecken und unter verschiedenen Wetterbedingungen fliegen zu können, Bilder mit hoher Auflösung aufzunehmen, die dann auch für die archäologische Dokumentation nutzbar und damit auch anderen Verfahren vorzuziehen sind. Da sich die UAVs in Bezug auf Entwicklungsgrad und Zuverlässigkeit rasant verbessert haben, haben ihre Möglichkeiten zur Unterstützung der archäologischen Forschung in jüngster Zeit großes Interesse geweckt, insbesondere bei der Dokumentation von Stätten, Denkmälern und Ausgrabungen. In dieser Fallstudie werden mehrere Kampagnen von Luftaufnahmen mit einer UAV-Wärmebildkamera auf einem archäologischen Gebiet durchgeführt. Nach der Bildaufufnahme die Farb- und Wärmebilder in einer entsprechenden Software verarbeitet. Der nächste Schritt wird die Verknüpfung der Bilder sein, um ein genaues und georeferenziertes 3D- und Netzmodell des vermessenden Bereichs zu erstellen. Anschließend werden farbige und thermische Orthophoto-Mosaike sowie digitale Oberflächenmodelle (DSM) für die Dokumentation abgeleitet. Die Datensätze von Wärme- und Farbbildern werden gesammelt und verglichen, um archäologische Überreste auf und unter dem Boden zu erkennen

SeaVipers - Computer Vision and Inertial Position Reference Sensor System (CVIPRSS)

This work describes the design and development of an optical, Computer Vision (CV) based sensor for use as a Position Reference System (PRS) in Dynamic Positioning (DP). Using a combination of robotics and CV techniques, the sensor provides range and heading information to a selected reference object. The proposed optical system is superior to existing ones because it does not depend upon special reflectors nor does it require a lengthy set-up time. This system, the Computer Vision and Inertial Position Reference Sensor System (CVIPRSS, pronounced \nickname), combines a laser rangefinder, infrared camera, and a pan--tilt unit with the robust TLD (Tracking--Learning--Detection) object tracker. In this work, a \nickname ~prototype is evaluated, showing promising results as viable PRS with research, commercial, and industrial applications