Smart environment monitoring through micro unmanned aerial vehicles

In recent years, the improvements of small-scale Unmanned Aerial Vehicles (UAVs) in terms of flight time, automatic control, and remote transmission are promoting the development of a wide range of practical applications. In aerial video surveillance, the monitoring of broad areas still has many challenges due to the achievement of different tasks in real-time, including mosaicking, change detection, and object detection. In this thesis work, a small-scale UAV based vision system to maintain regular surveillance over target areas is proposed. The system works in two modes. The first mode allows to monitor an area of interest by performing several flights. During the first flight, it creates an incremental geo-referenced mosaic of an area of interest and classifies all the known elements (e.g., persons) found on the ground by an improved Faster R-CNN architecture previously trained. In subsequent reconnaissance flights, the system searches for any changes (e.g., disappearance of persons) that may occur in the mosaic by a histogram equalization and RGB-Local Binary Pattern (RGB-LBP) based algorithm. If present, the mosaic is updated. The second mode, allows to perform a real-time classification by using, again, our improved Faster R-CNN model, useful for time-critical operations. Thanks to different design features, the system works in real-time and performs mosaicking and change detection tasks at low-altitude, thus allowing the classification even of small objects. The proposed system was tested by using the whole set of challenging video sequences contained in the UAV Mosaicking and Change Detection (UMCD) dataset and other public datasets. The evaluation of the system by well-known performance metrics has shown remarkable results in terms of mosaic creation and updating, as well as in terms of change detection and object detection

Mapping and classification of ecologically sensitive marine habitats using unmanned aerial vehicle (UAV) imagery and object-based image analysis (OBIA)

Nowadays, emerging technologies, such as long-range transmitters, increasingly miniaturized components for positioning, and enhanced imaging sensors, have led to an upsurge in the availability of new ecological applications for remote sensing based on unmanned aerial vehicles (UAVs), sometimes referred to as “drones”. In fact, structure-from-motion (SfM) photogrammetry coupled with imagery acquired by UAVs offers a rapid and inexpensive tool to produce high-resolution orthomosaics, giving ecologists a new way for responsive, timely, and cost-effective monitoring of ecological processes. Here, we adopted a lightweight quadcopter as an aerial survey tool and object-based image analysis (OBIA) workflow to demonstrate the strength of such methods in producing very high spatial resolution maps of sensitive marine habitats. Therefore, three different coastal environments were mapped using the autonomous flight capability of a lightweight UAV equipped with a fully stabilized consumer-grade RGB digital camera. In particular we investigated a Posidonia oceanica seagrass meadow, a rocky coast with nurseries for juvenile fish, and two sandy areas showing biogenic reefs of Sabelleria alveolata. We adopted, for the first time, UAV-based raster thematic maps of these key coastal habitats, produced after OBIA classification, as a new method for fine-scale, low-cost, and time saving characterization of sensitive marine environments which may lead to a more effective and efficient monitoring and management of natural resource

Modélisation tridimensionnelle précise de l'environnement à l’aide des systèmes de photogrammétrie embarqués sur drones

Abstract : Images acquired from unmanned aerial vehicles (UAVs) can provide data with unprecedented spatial and temporal resolution for three-dimensional (3D) modeling. Solutions developed for this purpose are mainly operating based on photogrammetry concepts, namely UAV-Photogrammetry Systems (UAV-PS). Such systems are used in applications where both geospatial and visual information of the environment is required. These applications include, but are not limited to, natural resource management such as precision agriculture, military and police-related services such as traffic-law enforcement, precision engineering such as infrastructure inspection, and health services such as epidemic emergency management. UAV-photogrammetry systems can be differentiated based on their spatial characteristics in terms of accuracy and resolution. That is some applications, such as precision engineering, require high-resolution and high-accuracy information of the environment (e.g. 3D modeling with less than one centimeter accuracy and resolution). In other applications, lower levels of accuracy might be sufficient, (e.g. wildlife management needing few decimeters of resolution). However, even in those applications, the specific characteristics of UAV-PSs should be well considered in the steps of both system development and application in order to yield satisfying results. In this regard, this thesis presents a comprehensive review of the applications of unmanned aerial imagery, where the objective was to determine the challenges that remote-sensing applications of UAV systems currently face. This review also allowed recognizing the specific characteristics and requirements of UAV-PSs, which are mostly ignored or not thoroughly assessed in recent studies. Accordingly, the focus of the first part of this thesis is on exploring the methodological and experimental aspects of implementing a UAV-PS. The developed system was extensively evaluated for precise modeling of an open-pit gravel mine and performing volumetric-change measurements. This application was selected for two main reasons. Firstly, this case study provided a challenging environment for 3D modeling, in terms of scale changes, terrain relief variations as well as structure and texture diversities. Secondly, open-pit-mine monitoring demands high levels of accuracy, which justifies our efforts to improve the developed UAV-PS to its maximum capacities. The hardware of the system consisted of an electric-powered helicopter, a high-resolution digital camera, and an inertial navigation system. The software of the system included the in-house programs specifically designed for camera calibration, platform calibration, system integration, onboard data acquisition, flight planning and ground control point (GCP) detection. The detailed features of the system are discussed in the thesis, and solutions are proposed in order to enhance the system and its photogrammetric outputs. The accuracy of the results was evaluated under various mapping conditions, including direct georeferencing and indirect georeferencing with different numbers, distributions and types of ground control points. Additionally, the effects of imaging configuration and network stability on modeling accuracy were assessed. The second part of this thesis concentrates on improving the techniques of sparse and dense reconstruction. The proposed solutions are alternatives to traditional aerial photogrammetry techniques, properly adapted to specific characteristics of unmanned, low-altitude imagery. Firstly, a method was developed for robust sparse matching and epipolar-geometry estimation. The main achievement of this method was its capacity to handle a very high percentage of outliers (errors among corresponding points) with remarkable computational efficiency (compared to the state-of-the-art techniques). Secondly, a block bundle adjustment (BBA) strategy was proposed based on the integration of intrinsic camera calibration parameters as pseudo-observations to Gauss-Helmert model. The principal advantage of this strategy was controlling the adverse effect of unstable imaging networks and noisy image observations on the accuracy of self-calibration. The sparse implementation of this strategy was also performed, which allowed its application to data sets containing a lot of tie points. Finally, the concepts of intrinsic curves were revisited for dense stereo matching. The proposed technique could achieve a high level of accuracy and efficiency by searching only through a small fraction of the whole disparity search space as well as internally handling occlusions and matching ambiguities. These photogrammetric solutions were extensively tested using synthetic data, close-range images and the images acquired from the gravel-pit mine. Achieving absolute 3D mapping accuracy of 11±7 mm illustrated the success of this system for high-precision modeling of the environment.Résumé : Les images acquises à l’aide d’aéronefs sans pilote (ASP) permettent de produire des données de résolutions spatiales et temporelles uniques pour la modélisation tridimensionnelle (3D). Les solutions développées pour ce secteur d’activité sont principalement basées sur des concepts de photogrammétrie et peuvent être identifiées comme des systèmes photogrammétriques embarqués sur aéronefs sans pilote (SP-ASP). Ils sont utilisés dans plusieurs applications environnementales où l’information géospatiale et visuelle est essentielle. Ces applications incluent notamment la gestion des ressources naturelles (ex. : agriculture de précision), la sécurité publique et militaire (ex. : gestion du trafic), les services d’ingénierie (ex. : inspection de bâtiments) et les services de santé publique (ex. : épidémiologie et gestion des risques). Les SP-ASP peuvent être subdivisés en catégories selon les besoins en termes de précision et de résolution. En effet, dans certains cas, tel qu’en ingénierie, l’information sur l’environnement doit être de haute précision et de haute résolution (ex. : modélisation 3D avec une précision et une résolution inférieure à un centimètre). Pour d’autres applications, tel qu’en gestion de la faune sauvage, des niveaux de précision et de résolution moindres peut être suffisants (ex. : résolution de l’ordre de quelques décimètres). Cependant, même dans ce type d’applications les caractéristiques des SP-ASP devraient être prises en considération dans le développement des systèmes et dans leur utilisation, et ce, pour atteindre les résultats visés. À cet égard, cette thèse présente une revue exhaustive des applications de l’imagerie aérienne acquise par ASP et de déterminer les challenges les plus courants. Cette étude a également permis d’établir les caractéristiques et exigences spécifiques des SP-ASP qui sont généralement ignorées ou partiellement discutées dans les études récentes. En conséquence, la première partie de cette thèse traite des aspects méthodologiques et d’expérimentation de la mise en place d’un SP-ASP. Le système développé a été évalué pour la modélisation précise d’une gravière et utilisé pour réaliser des mesures de changement volumétrique. Cette application a été retenue pour deux raisons principales. Premièrement, ce type de milieu fournit un environnement difficile pour la modélisation, et ce, en termes de changement d’échelle, de changement de relief du terrain ainsi que la grande diversité de structures et de textures. Deuxièment, le suivi de mines à ciel ouvert exige un niveau de précision élevé, ce qui justifie les efforts déployés pour mettre au point un SP-ASP de haute précision. Les composantes matérielles du système consistent en un ASP à propulsion électrique de type hélicoptère, d’une caméra numérique à haute résolution ainsi qu’une station inertielle. La composante logicielle est composée de plusieurs programmes développés particulièrement pour calibrer la caméra et la plateforme, intégrer les systèmes, enregistrer les données, planifier les paramètres de vol et détecter automatiquement les points de contrôle au sol. Les détails complets du système sont abordés dans la thèse et des solutions sont proposées afin d’améliorer le système et la qualité des données photogrammétriques produites. La précision des résultats a été évaluée sous diverses conditions de cartographie, incluant le géoréférencement direct et indirect avec un nombre, une répartition et des types de points de contrôle variés. De plus, les effets de la configuration des images et la stabilité du réseau sur la précision de la modélisation ont été évalués. La deuxième partie de la thèse porte sur l’amélioration des techniques de reconstruction éparse et dense. Les solutions proposées sont des alternatives aux techniques de photogrammétrie aérienne traditionnelle et adaptée aux caractéristiques particulières de l’imagerie acquise à basse altitude par ASP. Tout d’abord, une méthode robuste de correspondance éparse et d’estimation de la géométrie épipolaire a été développée. L’élément clé de cette méthode est sa capacité à gérer le pourcentage très élevé des valeurs aberrantes (erreurs entre les points correspondants) avec une efficacité de calcul remarquable en comparaison avec les techniques usuelles. Ensuite, une stratégie d’ajustement de bloc basée sur l’intégration de pseudoobservations du modèle Gauss-Helmert a été proposée. Le principal avantage de cette stratégie consistait à contrôler les effets négatifs du réseau d’images instable et des images bruitées sur la précision de l’autocalibration. Une implémentation éparse de cette stratégie a aussi été réalisée, ce qui a permis de traiter des jeux de données contenant des millions de points de liaison. Finalement, les concepts de courbes intrinsèques ont été revisités pour l’appariement stéréo dense. La technique proposée pourrait atteindre un haut niveau de précision et d’efficacité en recherchant uniquement dans une petite portion de l’espace de recherche des disparités ainsi qu’en traitant les occlusions et les ambigüités d’appariement. Ces solutions photogrammétriques ont été largement testées à l’aide de données synthétiques, d’images à courte portée ainsi que celles acquises sur le site de la gravière. Le système a démontré sa capacité a modélisation dense de l’environnement avec une très haute exactitude en atteignant une précision 3D absolue de l’ordre de 11±7 mm

Point Cloud Generation from Aerial Image Data Acquired by a Quadrocopter Type Micro Unmanned Aerial Vehicle and a Digital Still Camera

The objective of this investigation was to develop and investigate methods for point cloud generation by image matching using aerial image data collected by quadrocopter type micro unmanned aerial vehicle (UAV) imaging systems. Automatic generation of high-quality, dense point clouds from digital images by image matching is a recent, cutting-edge step forward in digital photogrammetric technology. The major components of the system for point cloud generation are a UAV imaging system, an image data collection process using high image overlaps, and post-processing with image orientation and point cloud generation. Two post-processing approaches were developed: one of the methods is based on Bae Systems’ SOCET SET classical commercial photogrammetric software and another is built using Microsoft®’s Photosynth™ service available in the Internet. Empirical testing was carried out in two test areas. Photosynth processing showed that it is possible to orient the images and generate point clouds fully automatically without any a priori orientation information or interactive work. The photogrammetric processing line provided dense and accurate point clouds that followed the theoretical principles of photogrammetry, but also some artifacts were detected. The point clouds from the Photosynth processing were sparser and noisier, which is to a large extent due to the fact that the method is not optimized for dense point cloud generation. Careful photogrammetric processing with self-calibration is required to achieve the highest accuracy. Our results demonstrate the high performance potential of the approach and that with rigorous processing it is possible to reach results that are consistent with theory. We also point out several further research topics. Based on theoretical and empirical results, we give recommendations for properties of imaging sensor, data collection and processing of UAV image data to ensure accurate point cloud generation

Ffau—framework for fully autonomous uavs

Nr. 024539 (POCI-01-0247-FEDER-024539) under grant agreement No 783221 UID/EEA/00066/2019Unmanned Aerial Vehicles (UAVs), although hardly a new technology, have recently gained a prominent role in many industries being widely used not only among enthusiastic consumers, but also in high demanding professional situations, and will have a massive societal impact over the coming years. However, the operation of UAVs is fraught with serious safety risks, such as collisions with dynamic obstacles (birds, other UAVs, or randomly thrown objects). These collision scenarios are complex to analyze in real-time, sometimes being computationally impossible to solve with existing State of the Art (SoA) algorithms, making the use of UAVs an operational hazard and therefore significantly reducing their commercial applicability in urban environments. In this work, a conceptual framework for both stand-alone and swarm (networked) UAVs is introduced, with a focus on the architectural requirements of the collision avoidance subsystem to achieve acceptable levels of safety and reliability. The SoA principles for collision avoidance against stationary objects are reviewed and a novel approach is described, using deep learning techniques to solve the computational intensive problem of real-time collision avoidance with dynamic objects. The proposed framework includes a web-interface allowing the full control of UAVs as remote clients with a supervisor cloud-based platform. The feasibility of the proposed approach was demonstrated through experimental tests using a UAV, developed from scratch using the proposed framework. Test flight results are presented for an autonomous UAV monitored from multiple countries across the world.publishersversionpublishe

Automated Image Registration And Mosaicking For Multi-Sensor Images Acquired By A Miniature Unmanned Aerial Vehicle Platform

Algorithms for automatic image registration and mosaicking are developed for a miniature Unmanned Aerial Vehicle (MINI-UAV) platform, assembled by Air-O-Space International (AOSI) L.L.C.. Three cameras onboard this MINI-UAV platform acquire images in a single frame simultaneously at green (550nm), red (650 nm), and near infrared (820nm) wavelengths, but with shifting and rotational misalignment. The area-based method is employed in the developed algorithms for control point detection, which is applicable when no prominent feature details are present in image scenes. Because the three images to be registered have different spectral characteristics, region of interest determination and control point selection are the two key steps that ensure the quality of control points. Affine transformation is adopted for spatial transformation, followed by bilinear interpolation for image resampling. Mosaicking is conducted between adjacent frames after three-band co-registration. Pre-introducing the rotation makes the area-based method feasible when the rotational misalignment cannot be ignored. The algorithms are tested on three image sets collected at Stennis Space Center, Greenwood, and Oswalt in Mississippi. Manual evaluation confirms the effectiveness of the developed algorithms. The codes are converted into a software package, which is executable under the Microsoft Windows environment of personal computer platforms without the requirement of MATLAB or other special software support for commercial-off-the-shelf (COTS) product. The near real-time decision-making support is achievable with final data after its installation into the ground control station. The final products are color-infrared (CIR) composite and normalized difference vegetation index (NDVI) images, which are used in agriculture, forestry, and environmental monitoring

Using Linear Features for Aerial Image Sequence Mosaiking

With recent advances in sensor technology and digital image processing techniques, automatic image mosaicking has received increased attention in a variety of geospatial applications, ranging from panorama generation and video surveillance to image based rendering. The geometric transformation used to link images in a mosaic is the subject of image orientation, a fundamental photogrammetric task that represents a major research area in digital image analysis. It involves the determination of the parameters that express the location and pose of a camera at the time it captured an image. In aerial applications the typical parameters comprise two translations (along the x and y coordinates) and one rotation (rotation about the z axis). Orientation typically proceeds by extracting from an image control points, i.e. points with known coordinates. Salient points such as road intersections, and building corners are commonly used to perform this task. However, such points may contain minimal information other than their radiometric uniqueness, and, more importantly, in some areas they may be impossible to obtain (e.g. in rural and arid areas). To overcome this problem we introduce an alternative approach that uses linear features such as roads and rivers for image mosaicking. Such features are identified and matched to their counterparts in overlapping imagery. Our matching approach uses critical points (e.g. breakpoints) of linear features and the information conveyed by them (e.g. local curvature values and distance metrics) to match two such features and orient the images in which they are depicted. In this manner we orient overlapping images by comparing breakpoint representations of complete or partial linear features depicted in them. By considering broader feature metrics (instead of single points) in our matching scheme we aim to eliminate the effect of erroneous point matches in image mosaicking. Our approach does not require prior approximate parameters, which are typically an essential requirement for successful convergence of point matching schemes. Furthermore, we show that large rotation variations about the z-axis may be recovered. With the acquired orientation parameters, image sequences are mosaicked. Experiments with synthetic aerial image sequences are included in this thesis to demonstrate the performance of our approach