Smart environment monitoring through micro unmanned aerial vehicles

In recent years, the improvements of small-scale Unmanned Aerial Vehicles (UAVs) in terms of flight time, automatic control, and remote transmission are promoting the development of a wide range of practical applications. In aerial video surveillance, the monitoring of broad areas still has many challenges due to the achievement of different tasks in real-time, including mosaicking, change detection, and object detection. In this thesis work, a small-scale UAV based vision system to maintain regular surveillance over target areas is proposed. The system works in two modes. The first mode allows to monitor an area of interest by performing several flights. During the first flight, it creates an incremental geo-referenced mosaic of an area of interest and classifies all the known elements (e.g., persons) found on the ground by an improved Faster R-CNN architecture previously trained. In subsequent reconnaissance flights, the system searches for any changes (e.g., disappearance of persons) that may occur in the mosaic by a histogram equalization and RGB-Local Binary Pattern (RGB-LBP) based algorithm. If present, the mosaic is updated. The second mode, allows to perform a real-time classification by using, again, our improved Faster R-CNN model, useful for time-critical operations. Thanks to different design features, the system works in real-time and performs mosaicking and change detection tasks at low-altitude, thus allowing the classification even of small objects. The proposed system was tested by using the whole set of challenging video sequences contained in the UAV Mosaicking and Change Detection (UMCD) dataset and other public datasets. The evaluation of the system by well-known performance metrics has shown remarkable results in terms of mosaic creation and updating, as well as in terms of change detection and object detection

Resolving Fine-Scale Surface Features on Polar Sea Ice: A First Assessment of UAS Photogrammetry Without Ground Control

Mapping landfast sea ice at a fine spatial scale is not only meaningful for geophysical study, but is also of benefit for providing information about human activities upon it. The combination of unmanned aerial systems (UAS) with structure from motion (SfM) methods have already revolutionized the current close-range Earth observation paradigm. To test their feasibility in characterizing the properties and dynamics of fast ice, three flights were carried out in the 2016–2017 austral summer during the 33rd Chinese National Antarctic Expedition (CHINARE), focusing on the area of the Prydz Bay in East Antarctica. Three-dimensional models and orthomosaics from three sorties were constructed from a total of 205 photos using Agisoft PhotoScan software. Logistical challenges presented by the terrain precluded the deployment of a dedicated ground control network; however, it was still possible to indirectly assess the performance of the photogrammetric products through an analysis of the statistics of the matching network, bundle adjustment, and Monte-Carlo simulation. Our results show that the matching networks are quite strong, given a sufficient number of feature points (mostly > 20,000) or valid matches (mostly > 1000). The largest contribution to the total error using our direct georeferencing approach is attributed to inaccuracies in the onboard position and orientation system (POS) records, especially in the vehicle height and yaw angle. On one hand, the 3D precision map reveals that planimetric precision is usually about one-third of the vertical estimate (typically 20 cm in the network centre). On the other hand, shape-only errors account for less than 5% for the X and Y dimensions and 20% for the Z dimension. To further illustrate the UAS’s capability, six representative surface features are selected and interpreted by sea ice experts. Finally, we offer pragmatic suggestions and guidelines for planning future UAS-SfM surveys without the use of ground control. The work represents a pioneering attempt to comprehensively assess UAS-SfM survey capability in fast ice environments, and could serve as a reference for future improvements

Quantifying spatial uncertainties in structure from motion snow depth mapping with drones in an alpine environment

Due to the heterogeneous nature of alpine snow distribution, advances in hydrological monitoring and forecasting for water resource management require an increase in the frequency, spatial resolution and coverage of field observations. Such detailed snow information is also needed to foster advances in our understanding of how snowpack affects local ecology and geomorphology. Although recent use of structure-from-motion multi-view stereo (SFM-MVS) 3D reconstruction techniques combined with aerial image collection using drones has shown promising potential to provide higher spatial and temporal resolution snow depth data for snowpack monitoring, there still remain challenges to produce high-quality data with this approach. These challenges, which include differentiating observations from noise and overcoming biases in the elevation data, are inherent in digital elevation model (DEM) differencing. A key issue to address these challenges is our ability to quantify measurement uncertainties in the SFM-MVS snow depths which can vary in space and time. The purpose of this thesis was to develop data-driven approaches for spatially quantifying, characterizing and reducing uncertainties in SFM-MVS snow depth mapping in alpine areas. Overall, this thesis provides a general framework for performing a detailed analysis of the spatial pattern of SFM-MVS snow depth uncertainties, as well as provides an approach for correction of snow depth errors due to changes in the sub-snow topography occurring between survey acquisition dates. It also contributes to the growing support of SFM-MVS combined with imagery acquired from drones as a suitable surveying technique for local scale snow distribution monitoring in alpine areas

Fast and accurate mosaicing techniques for aerial images of quasi-planar scenes

Image mosaicing aims to increase visual perception by composing data from separate images since a mosaic image provides a more powerful scene description. Gaining and maintaining situational awareness from image mosaics is important for both civil and military applications. Inspection of the urban areas suffering from natural disasters and examination of the large plantations are possible civil areas of utilization. For military applications, image mosaicing can provide critical information about enemy activities in wide areas. Although there are many studies in the literature that focus on creating real-time image mosaics for different applications, there is still room for improvement due to the need for faster and more accurate mosaicing for a variety of practical scenarios. In this thesis, novel techniques for creating fast and accurate aerial image mosaics of quasi-planar scenes are developed. First, a sequential mosaicing approach is proposed where all the past images intersecting the new image are used to estimate alignment of the new image. A tool from computer graphics, Separating Axis Theorem (SAT), is employed to detect image intersections. A new local affine refinement is introduced to provide global consistency throughout the mosaic. Second, a pose estimation based mosaicing technique is developed where the scene normal and the camera pose parameters are estimated through an Extended Kalman Filter (EKF). Mosaic is formed by using the homographies constructed from the estimated state vector. Using an EKF based approach provides a significant global consistency throughout the mosaic since all the parameters are updated by which error accumulations in the loop closing regions are compensated. Proposed algorithm also provides localization and attitude information of the camera which might be beneficial for robotics applications. Both methods are verified through several experiments and comparisons with some state-of-the-art algorithms are presented. Results show that the developed algorithms work successfully as intended

An Image-Based Real-Time Georeferencing Scheme for a UAV Based on a New Angular Parametrization

Simultaneous localization and mapping (SLAM) of a monocular projective camera installed on an unmanned aerial vehicle (UAV) is a challenging task in photogrammetry, computer vision, and robotics. This paper presents a novel real-time monocular SLAM solution for UAV applications. It is based on two steps: consecutive construction of the UAV path, and adjacent strip connection. Consecutive construction rapidly estimates the UAV path by sequentially connecting incoming images to a network of connected images. A multilevel pyramid matching is proposed for this step that contains a sub-window matching using high-resolution images. The sub-window matching increases the frequency of tie points by propagating locations of matched sub-windows that leads to a list of high-frequency tie points while keeping the execution time relatively low. A sparse bundle block adjustment (BBA) is employed to optimize the initial path by considering nuisance parameters. System calibration parameters with respect to global navigation satellite system (GNSS) and inertial navigation system (INS) are optionally considered in the BBA model for direct georeferencing. Ground control points and checkpoints are optionally included in the model for georeferencing and quality control. Adjacent strip connection is enabled by an overlap analysis to further improve connectivity of local networks. A novel angular parametrization based on spherical rotation coordinate system is presented to address the gimbal lock singularity of BBA. Our results suggest that the proposed scheme is a precise real-time monocular SLAM solution for a UAV.Peer reviewe