Smart environment monitoring through micro unmanned aerial vehicles

In recent years, the improvements of small-scale Unmanned Aerial Vehicles (UAVs) in terms of flight time, automatic control, and remote transmission are promoting the development of a wide range of practical applications. In aerial video surveillance, the monitoring of broad areas still has many challenges due to the achievement of different tasks in real-time, including mosaicking, change detection, and object detection. In this thesis work, a small-scale UAV based vision system to maintain regular surveillance over target areas is proposed. The system works in two modes. The first mode allows to monitor an area of interest by performing several flights. During the first flight, it creates an incremental geo-referenced mosaic of an area of interest and classifies all the known elements (e.g., persons) found on the ground by an improved Faster R-CNN architecture previously trained. In subsequent reconnaissance flights, the system searches for any changes (e.g., disappearance of persons) that may occur in the mosaic by a histogram equalization and RGB-Local Binary Pattern (RGB-LBP) based algorithm. If present, the mosaic is updated. The second mode, allows to perform a real-time classification by using, again, our improved Faster R-CNN model, useful for time-critical operations. Thanks to different design features, the system works in real-time and performs mosaicking and change detection tasks at low-altitude, thus allowing the classification even of small objects. The proposed system was tested by using the whole set of challenging video sequences contained in the UAV Mosaicking and Change Detection (UMCD) dataset and other public datasets. The evaluation of the system by well-known performance metrics has shown remarkable results in terms of mosaic creation and updating, as well as in terms of change detection and object detection

RGB2LIDAR: Towards Solving Large-Scale Cross-Modal Visual Localization

We study an important, yet largely unexplored problem of large-scale cross-modal visual localization by matching ground RGB images to a geo-referenced aerial LIDAR 3D point cloud (rendered as depth images). Prior works were demonstrated on small datasets and did not lend themselves to scaling up for large-scale applications. To enable large-scale evaluation, we introduce a new dataset containing over 550K pairs (covering 143 km^2 area) of RGB and aerial LIDAR depth images. We propose a novel joint embedding based method that effectively combines the appearance and semantic cues from both modalities to handle drastic cross-modal variations. Experiments on the proposed dataset show that our model achieves a strong result of a median rank of 5 in matching across a large test set of 50K location pairs collected from a 14km^2 area. This represents a significant advancement over prior works in performance and scale. We conclude with qualitative results to highlight the challenging nature of this task and the benefits of the proposed model. Our work provides a foundation for further research in cross-modal visual localization.Comment: ACM Multimedia 202

Geo-Information Harvesting from Social Media Data

As unconventional sources of geo-information, massive imagery and text messages from open platforms and social media form a temporally quasi-seamless, spatially multi-perspective stream, but with unknown and diverse quality. Due to its complementarity to remote sensing data, geo-information from these sources offers promising perspectives, but harvesting is not trivial due to its data characteristics. In this article, we address key aspects in the field, including data availability, analysis-ready data preparation and data management, geo-information extraction from social media text messages and images, and the fusion of social media and remote sensing data. We then showcase some exemplary geographic applications. In addition, we present the first extensive discussion of ethical considerations of social media data in the context of geo-information harvesting and geographic applications. With this effort, we wish to stimulate curiosity and lay the groundwork for researchers who intend to explore social media data for geo-applications. We encourage the community to join forces by sharing their code and data.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin

Fine-grained traffic state estimation and visualisation

Tools for visualising the current traffic state are used by local authorities for strategic monitoring of the traffic network and by everyday users for planning their journey. Popular visualisations include those provided by Google Maps and by Inrix. Both employ a traffic lights colour-coding system, where roads on a map are coloured green if traffic is flowing normally and red or black if there is congestion. New sensor technology, especially from wireless sources, is allowing resolution down to lane level. A case study is reported in which a traffic micro-simulation test bed is used to generate high-resolution estimates. An interactive visualisation of the fine-grained traffic state is presented. The visualisation is demonstrated using Google Earth and affords the user a detailed three-dimensional view of the traffic state down to lane level in real time

A MODIS/ASTER Airborne Simulator (MASTER) Imagery for Urban Heat Island Research

abstract: Thermal imagery is widely used to quantify land surface temperatures to monitor the spatial extent and thermal intensity of the urban heat island (UHI) effect. Previous research has applied Landsat images, Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) images, Moderate Resolution Imaging Spectroradiometer (MODIS) images, and other coarse- to medium-resolution remotely sensed imagery to estimate surface temperature. These data are frequently correlated with vegetation, impervious surfaces, and temperature to quantify the drivers of the UHI effect. Because of the coarse- to medium-resolution of the thermal imagery, researchers are unable to correlate these temperature data with the more generally available high-resolution land cover classification, which are derived from high-resolution multispectral imagery. The development of advanced thermal sensors with very high-resolution thermal imagery such as the MODIS/ASTER airborne simulator (MASTER) has investigators quantifying the relationship between detailed land cover and land surface temperature. While this is an obvious next step, the published literature, i.e., the MASTER data, are often used to discriminate burned areas, assess fire severity, and classify urban land cover. Considerably less attention is given to use MASTER data in the UHI research. We demonstrate here that MASTER data in combination with high-resolution multispectral data has made it possible to monitor and model the relationship between temperature and detailed land cover such as building rooftops, residential street pavements, and parcel-based landscaping. Here, we report on data sources to conduct this type of UHI research and endeavor to intrigue researchers and scientists such that high-resolution airborne thermal imagery is used to further explore the UHI effect

A MODIS/ASTER airborne simulator (MASTER) imagery for urban heat island research

Thermal imagery is widely used to quantify land surface temperatures to monitor the spatial extent and thermal intensity of the urban heat island (UHI) effect. Previous research has applied Landsat images, Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) images, Moderate Resolution Imaging Spectroradiometer (MODIS) images, and other coarse- to medium-resolution remotely sensed imagery to estimate surface temperature. These data are frequently correlated with vegetation, impervious surfaces, and temperature to quantify the drivers of the UHI effect. Because of the coarse- to medium-resolution of the thermal imagery, researchers are unable to correlate these temperature data with the more generally available high-resolution land cover classification, which are derived from high-resolution multispectral imagery. The development of advanced thermal sensors with very high-resolution thermal imagery such as the MODIS/ASTER airborne simulator (MASTER) has investigators quantifying the relationship between detailed land cover and land surface temperature. While this is an obvious next step, the published literature, i.e., the MASTER data, are often used to discriminate burned areas, assess fire severity, and classify urban land cover. Considerably less attention is given to use MASTER data in the UHI research. We demonstrate here that MASTER data in combination with high-resolution multispectral data has made it possible to monitor and model the relationship between temperature and detailed land cover such as building rooftops, residential street pavements, and parcel-based landscaping. Here, we report on data sources to conduct this type of UHI research and endeavor to intrigue researchers and scientists such that high-resolution airborne thermal imagery is used to further explore the UHI effect

Cross-View Visual Geo-Localization for Outdoor Augmented Reality

Precise estimation of global orientation and location is critical to ensure a compelling outdoor Augmented Reality (AR) experience. We address the problem of geo-pose estimation by cross-view matching of query ground images to a geo-referenced aerial satellite image database. Recently, neural network-based methods have shown state-of-the-art performance in cross-view matching. However, most of the prior works focus only on location estimation, ignoring orientation, which cannot meet the requirements in outdoor AR applications. We propose a new transformer neural network-based model and a modified triplet ranking loss for joint location and orientation estimation. Experiments on several benchmark cross-view geo-localization datasets show that our model achieves state-of-the-art performance. Furthermore, we present an approach to extend the single image query-based geo-localization approach by utilizing temporal information from a navigation pipeline for robust continuous geo-localization. Experimentation on several large-scale real-world video sequences demonstrates that our approach enables high-precision and stable AR insertion.Comment: IEEE VR 202