Deep Learning-Based Object Detection in Maritime Unmanned Aerial Vehicle Imagery: Review and Experimental Comparisons

With the advancement of maritime unmanned aerial vehicles (UAVs) and deep learning technologies, the application of UAV-based object detection has become increasingly significant in the fields of maritime industry and ocean engineering. Endowed with intelligent sensing capabilities, the maritime UAVs enable effective and efficient maritime surveillance. To further promote the development of maritime UAV-based object detection, this paper provides a comprehensive review of challenges, relative methods, and UAV aerial datasets. Specifically, in this work, we first briefly summarize four challenges for object detection on maritime UAVs, i.e., object feature diversity, device limitation, maritime environment variability, and dataset scarcity. We then focus on computational methods to improve maritime UAV-based object detection performance in terms of scale-aware, small object detection, view-aware, rotated object detection, lightweight methods, and others. Next, we review the UAV aerial image/video datasets and propose a maritime UAV aerial dataset named MS2ship for ship detection. Furthermore, we conduct a series of experiments to present the performance evaluation and robustness analysis of object detection methods on maritime datasets. Eventually, we give the discussion and outlook on future works for maritime UAV-based object detection. The MS2ship dataset is available at \href{https://github.com/zcj234/MS2ship}{https://github.com/zcj234/MS2ship}.Comment: 32 pages, 18 figure

Artificial Intelligence Applications for Drones Navigation in GPS-denied or degraded Environments

L'abstract è presente nell'allegato / the abstract is in the attachmen

Contributions to improve the technologies supporting unmanned aircraft operations

Mención Internacional en el título de doctorUnmanned Aerial Vehicles (UAVs), in their smaller versions known as drones, are becoming increasingly important in today's societies. The systems that make them up present a multitude of challenges, of which error can be considered the common denominator. The perception of the environment is measured by sensors that have errors, the models that interpret the information and/or define behaviors are approximations of the world and therefore also have errors. Explaining error allows extending the limits of deterministic models to address real-world problems. The performance of the technologies embedded in drones depends on our ability to understand, model, and control the error of the systems that integrate them, as well as new technologies that may emerge. Flight controllers integrate various subsystems that are generally dependent on other systems. One example is the guidance systems. These systems provide the engine's propulsion controller with the necessary information to accomplish a desired mission. For this purpose, the flight controller is made up of a control law for the guidance system that reacts to the information perceived by the perception and navigation systems. The error of any of the subsystems propagates through the ecosystem of the controller, so the study of each of them is essential. On the other hand, among the strategies for error control are state-space estimators, where the Kalman filter has been a great ally of engineers since its appearance in the 1960s. Kalman filters are at the heart of information fusion systems, minimizing the error covariance of the system and allowing the measured states to be filtered and estimated in the absence of observations. State Space Models (SSM) are developed based on a set of hypotheses for modeling the world. Among the assumptions are that the models of the world must be linear, Markovian, and that the error of their models must be Gaussian. In general, systems are not linear, so linearization are performed on models that are already approximations of the world. In other cases, the noise to be controlled is not Gaussian, but it is approximated to that distribution in order to be able to deal with it. On the other hand, many systems are not Markovian, i.e., their states do not depend only on the previous state, but there are other dependencies that state space models cannot handle. This thesis deals a collection of studies in which error is formulated and reduced. First, the error in a computer vision-based precision landing system is studied, then estimation and filtering problems from the deep learning approach are addressed. Finally, classification concepts with deep learning over trajectories are studied. The first case of the collection xviiistudies the consequences of error propagation in a machine vision-based precision landing system. This paper proposes a set of strategies to reduce the impact on the guidance system, and ultimately reduce the error. The next two studies approach the estimation and filtering problem from the deep learning approach, where error is a function to be minimized by learning. The last case of the collection deals with a trajectory classification problem with real data. This work completes the two main fields in deep learning, regression and classification, where the error is considered as a probability function of class membership.Los vehículos aéreos no tripulados (UAV) en sus versiones de pequeño tamaño conocidos como drones, van tomando protagonismo en las sociedades actuales. Los sistemas que los componen presentan multitud de retos entre los cuales el error se puede considerar como el denominador común. La percepción del entorno se mide mediante sensores que tienen error, los modelos que interpretan la información y/o definen comportamientos son aproximaciones del mundo y por consiguiente también presentan error. Explicar el error permite extender los límites de los modelos deterministas para abordar problemas del mundo real. El rendimiento de las tecnologías embarcadas en los drones, dependen de nuestra capacidad de comprender, modelar y controlar el error de los sistemas que los integran, así como de las nuevas tecnologías que puedan surgir. Los controladores de vuelo integran diferentes subsistemas los cuales generalmente son dependientes de otros sistemas. Un caso de esta situación son los sistemas de guiado. Estos sistemas son los encargados de proporcionar al controlador de los motores información necesaria para cumplir con una misión deseada. Para ello se componen de una ley de control de guiado que reacciona a la información percibida por los sistemas de percepción y navegación. El error de cualquiera de estos sistemas se propaga por el ecosistema del controlador siendo vital su estudio. Por otro lado, entre las estrategias para abordar el control del error se encuentran los estimadores en espacios de estados, donde el filtro de Kalman desde su aparición en los años 60, ha sido y continúa siendo un gran aliado para los ingenieros. Los filtros de Kalman son el corazón de los sistemas de fusión de información, los cuales minimizan la covarianza del error del sistema, permitiendo filtrar los estados medidos y estimarlos cuando no se tienen observaciones. Los modelos de espacios de estados se desarrollan en base a un conjunto de hipótesis para modelar el mundo. Entre las hipótesis se encuentra que los modelos del mundo han de ser lineales, markovianos y que el error de sus modelos ha de ser gaussiano. Generalmente los sistemas no son lineales por lo que se realizan linealizaciones sobre modelos que a su vez ya son aproximaciones del mundo. En otros casos el ruido que se desea controlar no es gaussiano, pero se aproxima a esta distribución para poder abordarlo. Por otro lado, multitud de sistemas no son markovianos, es decir, sus estados no solo dependen del estado anterior, sino que existen otras dependencias que los modelos de espacio de estados no son capaces de abordar. Esta tesis aborda un compendio de estudios sobre los que se formula y reduce el error. En primer lugar, se estudia el error en un sistema de aterrizaje de precisión basado en visión por computador. Después se plantean problemas de estimación y filtrado desde la aproximación del aprendizaje profundo. Por último, se estudian los conceptos de clasificación con aprendizaje profundo sobre trayectorias. El primer caso del compendio estudia las consecuencias de la propagación del error de un sistema de aterrizaje de precisión basado en visión artificial. En este trabajo se propone un conjunto de estrategias para reducir el impacto sobre el sistema de guiado, y en última instancia reducir el error. Los siguientes dos estudios abordan el problema de estimación y filtrado desde la perspectiva del aprendizaje profundo, donde el error es una función que minimizar mediante aprendizaje. El último caso del compendio aborda un problema de clasificación de trayectorias con datos reales. Con este trabajo se completan los dos campos principales en aprendizaje profundo, regresión y clasificación, donde se plantea el error como una función de probabilidad de pertenencia a una clase.I would like to thank the Ministry of Science and Innovation for granting me the funding with reference PRE2018-086793, associated to the project TEC2017-88048-C2-2-R, which provide me the opportunity to carry out all my PhD. activities, including completing an international research internship.Programa de Doctorado en Ciencia y Tecnología Informática por la Universidad Carlos III de MadridPresidente: Antonio Berlanga de Jesús.- Secretario: Daniel Arias Medina.- Vocal: Alejandro Martínez Cav

Deep Learning Approach for UAV Visual Electrical Assets Inspection

The growth in the electrical demand by most countries around the world requires bigger and more complex energy systems, which leads to the requirement of having even more monitoring, inspection and maintenance of those systems. To respond to this need, inspection methods based on Unmanned Aerial Vehicles (UAV) have emerged which, when equipped with the appropriate sensors, allow a greater reduction of costs and risks and an increase in efficiency and effectiveness compared to traditional methods, such as inspection with foot patrols or helicopter-assisted. To make the inspection process more autonomous and reliable, most of the methods apply visual detection methods that use highly complex Deep Learning based algorithms and that require a very large computational power. This dissertation intends to present a system for inspection of electrical assets, able to be integrated onboard the UAV, based on Deep Learning, which allows to collect visual samples grouped and aggregated for each electrical asset detected. To this end, a perception system capable of detecting electrical insulators or structures, such as poles or transmission towers, was developed, using the Movidius Neural Compute Stick portable platform that is capable of processing lightweight object detection Convolutional Neural Networks, allowing a modular, low-cost system that meets real-time processing requirements. In addition to this perception system, an electrical asset monitoring system has been implemented that allows tracking and mapping each asset throughout the inspection process, based on the previous system’s detections and a UAV navigation system. Finally, an autonomous inspection system is proposed, which consists of a set of trajectories that allow an efficient application of the monitoring system along a power line, through the mapping of structures and the gathering of insulator samples around that structure.O grande crescimento da exigência elétrica pela maioria dos países por todo o mundo, requer que os sistemas de energia sejam maiores e mais complexos, o que conduz a uma maior necessidade de monitorização, inspeção e manutenção desses sistemas. Para responder a esta necessidade, surgiram métodos de inspeção baseados em Veículos Aéreos Não Tripulados (VANT) que, quando equipados com os sensores apropriados, permitem uma maior redução de custos e riscos e um grande aumento de eficiência e eficácia em comparação com os métodos tradicionais, como a inspeção com patrulhas pedonais ou assistida por helicóptero. Para tornar processo de inspeção mais autónomo e confiável, a maioria dos métodos realiza método de deteção visuais que utilizam algoritmos baseados em Deep Learning de elevada complexidade e que requerem um poder computacional muito grande. Nesta dissertação pretende-se apresentar um sistema de inspeção de ativos elétricos, para integração em VANTs, baseado em Apredizagem Profunda, que permite recolher amostras visuais agrupadas e agregadas por cada ativo elétrico detetado. Para tal foi desenvolvido um sistema de perceção capaz de detetar isoladores elétricos ou estruturas, como postes ou torres de transmissão, com recurso `a plataforma portátil Movidius Neural Compute Stick que ´e capaz de processar Redes Neuronais Convolucionais leves de deteção de objetos, permitindo assim um sistema modular, de baixo custo e que cumpre requisitos de processamento em tempo real. Para além deste sistema de perceção, foi implementado um sistema de monitorização de ativos elétricos que permite seguir e mapear cada ativo ao longo do processo de inspeção, com base nas deteções do sistema anterior e no sistema de navegação do VANT. Por fim, ´e proposto um sistema de inspeção autónomo que consiste num conjunto de trajetórias que permitem aplicar o sistema de monitorização de ativos elétricos ao longo de uma linha elétrica, através do mapeamento de estruturas e na recolha de amostras de isoladores em torno dessa estrutura

Collision Avoidance on Unmanned Aerial Vehicles using Deep Neural Networks

Unmanned Aerial Vehicles (UAVs), although hardly a new technology, have recently gained a prominent role in many industries, being widely used not only among enthusiastic consumers but also in high demanding professional situations, and will have a massive societal impact over the coming years. However, the operation of UAVs is full of serious safety risks, such as collisions with dynamic obstacles (birds, other UAVs, or randomly thrown objects). These collision scenarios are complex to analyze in real-time, sometimes being computationally impossible to solve with existing State of the Art (SoA) algorithms, making the use of UAVs an operational hazard and therefore significantly reducing their commercial applicability in urban environments. In this work, a conceptual framework for both stand-alone and swarm (networked) UAVs is introduced, focusing on the architectural requirements of the collision avoidance subsystem to achieve acceptable levels of safety and reliability. First, the SoA principles for collision avoidance against stationary objects are reviewed. Afterward, a novel image processing approach that uses deep learning and optical flow is presented. This approach is capable of detecting and generating escape trajectories against potential collisions with dynamic objects. Finally, novel models and algorithms combinations were tested, providing a new approach for the collision avoidance of UAVs using Deep Neural Networks. The feasibility of the proposed approach was demonstrated through experimental tests using a UAV, created from scratch using the framework developed.Os veículos aéreos não tripulados (VANTs), embora dificilmente considerados uma nova tecnologia, ganharam recentemente um papel de destaque em muitas indústrias, sendo amplamente utilizados não apenas por amadores, mas também em situações profissionais de alta exigência, sendo expectável um impacto social massivo nos próximos anos. No entanto, a operação de VANTs está repleta de sérios riscos de segurança, como colisões com obstáculos dinâmicos (pássaros, outros VANTs ou objetos arremessados). Estes cenários de colisão são complexos para analisar em tempo real, às vezes sendo computacionalmente impossível de resolver com os algoritmos existentes, tornando o uso de VANTs um risco operacional e, portanto, reduzindo significativamente a sua aplicabilidade comercial em ambientes citadinos. Neste trabalho, uma arquitectura conceptual para VANTs autônomos e em rede é apresentada, com foco nos requisitos arquitetônicos do subsistema de prevenção de colisão para atingir níveis aceitáveis de segurança e confiabilidade. Os estudos presentes na literatura para prevenção de colisão contra objectos estacionários são revistos e uma nova abordagem é descrita. Esta tecnica usa técnicas de aprendizagem profunda e processamento de imagem, para realizar a prevenção de colisões em tempo real com objetos móveis. Por fim, novos modelos e combinações de algoritmos são propostos, fornecendo uma nova abordagem para evitar colisões de VANTs usando Redes Neurais Profundas. A viabilidade da abordagem foi demonstrada através de testes experimentais utilizando um VANT, desenvolvido a partir da arquitectura apresentada

UAV or Drones for Remote Sensing Applications in GPS/GNSS Enabled and GPS/GNSS Denied Environments

The design of novel UAV systems and the use of UAV platforms integrated with robotic sensing and imaging techniques, as well as the development of processing workflows and the capacity of ultra-high temporal and spatial resolution data, have enabled a rapid uptake of UAVs and drones across several industries and application domains.This book provides a forum for high-quality peer-reviewed papers that broaden awareness and understanding of single- and multiple-UAV developments for remote sensing applications, and associated developments in sensor technology, data processing and communications, and UAV system design and sensing capabilities in GPS-enabled and, more broadly, Global Navigation Satellite System (GNSS)-enabled and GPS/GNSS-denied environments.Contributions include:UAV-based photogrammetry, laser scanning, multispectral imaging, hyperspectral imaging, and thermal imaging;UAV sensor applications; spatial ecology; pest detection; reef; forestry; volcanology; precision agriculture wildlife species tracking; search and rescue; target tracking; atmosphere monitoring; chemical, biological, and natural disaster phenomena; fire prevention, flood prevention; volcanic monitoring; pollution monitoring; microclimates; and land use;Wildlife and target detection and recognition from UAV imagery using deep learning and machine learning techniques;UAV-based change detection

Large-Scale Unmanned Aerial Systems Traffic Density Prediction and Management

In recent years, the applications of Unmanned Aerial Systems (UAS) has become more and more popular. We envision that in the near future, the complicated and high density UAS traffic will impose significant burden to air traffic management. Lot of works focus on the application development of individual Small Unmanned Aerial Systems (sUAS) or sUAS management Policy, however, the study of the UAS cluster behaviors such as forecasting and managing of the UAS traffic has generally not been addressed. In order to address the above issue, there is an urgent need to investigate three research directions. The first direction is to develop a high fidelity simulator for the UAS cluster behavior evaluation. The second direction to study real time trajectory planning algorithms to mitigate the high dense UAS traffic. The last direction is to investigate techniques that rapidly and accurately forecast the UAS traffic pattern in the future. In this thesis, we elaborate these three research topics and present a universal paradigm to predict and manage the traffic for the large-scale unmanned aerial systems. To enable the research in UAS traffic management and prediction, a Java based Multi-Agent Air Traffic and Resource Usage Simulation (MATRUS) framework is first developed. We use two types of UAS trajectories, Point-to-Point (P2P) and Man- hattan, as the case study to describe the capability of presented framework. Various communication and propagation models (i.e. log-distance-path loss) can be integrated with the framework to model the communication between UASs and base stations. The results show that MATRUS has the ability to evaluate different sUAS traffic management policies, and can provide insights on the relationships between air traf- fic and communication resource usage for further studies. Moreover, the framework can be extended to study the effect of sUAS Detect-and-Avoid (DAA) mechanisms, implement additional traffic management policies, and handle more complex traffic demands and geographical distributions. Based on the MATRUS framework, we propose a Sparse Represented Temporal- Spatial (SRTS) UAS trajectory planning algorithm. The SRTS algorithm allows the sUAS to avoid static no-fly areas (i.e. static obstacles) or other areas that have congested air traffic or communication traffic. The core functionality of the routing algorithm supports the instant refresh of the in-flight environment making it appropri- ate for highly dynamic air traffic scenarios. The characterization of the routing time and memory usage demonstrate that the SRTS algorithm outperforms a traditional Temporal-Spatial routing algorithm. The deep learning based approach has shown an outstanding success in many areas, we first investigated the possibility of applying the deep neural network in predicting the trajectory of a single vehicle in a given traffic scene. A new trajectory prediction model is developed, which allows information sharing among vehicles using a graph neural network. The prediction is based on the embedding feature, which is derived from multi-dimensional input sequences including the historical trajectory of target and neighboring vehicles, and their relative positions. Compared to other existing trajectory prediction methods, the proposed approach can reduce the pre- diction error by up to 50.00%. Then, we present a deep neural network model that extracts the features from both spatial and temporal domains to predict the UAS traffic density. In addition, a novel input representation of the future sUAS mission information is proposed. The pre-scheduled missions are categorized into 3 types according to their launching times. The results show that our presented model out- performs all of the baseline models. Meanwhile, the qualitative results demonstrate that our model can accurately predict the hot spot in the future traffic map

CMOS Image Sensors in Surveillance System Applications

Recent technology advances in CMOS image sensors (CIS) enable their utilization in the most demanding of surveillance fields, especially visual surveillance and intrusion detection in intelligent surveillance systems, aerial surveillance in war zones, Earth environmental surveillance by satellites in space monitoring, agricultural monitoring using wireless sensor networks and internet of things and driver assistance in automotive fields. This paper presents an overview of CMOS image sensor-based surveillance applications over the last decade by tabulating the design characteristics related to image quality such as resolution, frame rate, dynamic range, signal-to-noise ratio, and also processing technology. Different models of CMOS image sensors used in all applications have been surveyed and tabulated for every year and application.https://doi.org/10.3390/s2102048

Remote Sensing Data Compression

A huge amount of data is acquired nowadays by different remote sensing systems installed on satellites, aircrafts, and UAV. The acquired data then have to be transferred to image processing centres, stored and/or delivered to customers. In restricted scenarios, data compression is strongly desired or necessary. A wide diversity of coding methods can be used, depending on the requirements and their priority. In addition, the types and properties of images differ a lot, thus, practical implementation aspects have to be taken into account. The Special Issue paper collection taken as basis of this book touches on all of the aforementioned items to some degree, giving the reader an opportunity to learn about recent developments and research directions in the field of image compression. In particular, lossless and near-lossless compression of multi- and hyperspectral images still remains current, since such images constitute data arrays that are of extremely large size with rich information that can be retrieved from them for various applications. Another important aspect is the impact of lossless compression on image classification and segmentation, where a reasonable compromise between the characteristics of compression and the final tasks of data processing has to be achieved. The problems of data transition from UAV-based acquisition platforms, as well as the use of FPGA and neural networks, have become very important. Finally, attempts to apply compressive sensing approaches in remote sensing image processing with positive outcomes are observed. We hope that readers will find our book useful and interestin