Deep Learning-Based Object Detection in Maritime Unmanned Aerial Vehicle Imagery: Review and Experimental Comparisons

With the advancement of maritime unmanned aerial vehicles (UAVs) and deep learning technologies, the application of UAV-based object detection has become increasingly significant in the fields of maritime industry and ocean engineering. Endowed with intelligent sensing capabilities, the maritime UAVs enable effective and efficient maritime surveillance. To further promote the development of maritime UAV-based object detection, this paper provides a comprehensive review of challenges, relative methods, and UAV aerial datasets. Specifically, in this work, we first briefly summarize four challenges for object detection on maritime UAVs, i.e., object feature diversity, device limitation, maritime environment variability, and dataset scarcity. We then focus on computational methods to improve maritime UAV-based object detection performance in terms of scale-aware, small object detection, view-aware, rotated object detection, lightweight methods, and others. Next, we review the UAV aerial image/video datasets and propose a maritime UAV aerial dataset named MS2ship for ship detection. Furthermore, we conduct a series of experiments to present the performance evaluation and robustness analysis of object detection methods on maritime datasets. Eventually, we give the discussion and outlook on future works for maritime UAV-based object detection. The MS2ship dataset is available at \href{https://github.com/zcj234/MS2ship}{https://github.com/zcj234/MS2ship}.Comment: 32 pages, 18 figure

INTEROPERABILITY FOR MODELING AND SIMULATION IN MARITIME EXTENDED FRAMEWORK

This thesis reports on the most relevant researches performed during the years of the Ph.D. at the Genova University and within the Simulation Team. The researches have been performed according to M&S well known recognized standards. The studies performed on interoperable simulation cover all the environments of the Extended Maritime Framework, namely Sea Surface, Underwater, Air, Coast & Land, Space and Cyber Space. The applications cover both the civil and defence domain. The aim is to demonstrate the potential of M&S applications for the Extended Maritime Framework, applied to innovative unmanned vehicles as well as to traditional assets, human personnel included. A variety of techniques and methodology have been fruitfully applied in the researches, ranging from interoperable simulation, discrete event simulation, stochastic simulation, artificial intelligence, decision support system and even human behaviour modelling

Advantages and challenges of unmanned aerial vehicle autonomy in the Postheroic age

Over the past decade, unmanned aerial vehicles (UAVs) have revolutionized how the U.S. engages elusive militants in low-intensity conflicts by allowing the U.S. to project continuous military power without risking combat casualties. While UAV usage promises additional tactical advantages in future conflicts, little agreement exists regarding a strategic vision for UAV research and development, necessary for the U.S. to allocate limited resources among UAV development programs that address national security objectives. The present research makes the case for a future UAV technology evolutionary path leading to fully autonomous intelligence, surveillance, and reconnaissance (ISR)/strike UAV systems for the United States Air Force that are capable of sensing their environments through multiple modalities, recognizing patterns, and executing appropriate actions in response to their real-time analyses. The thesis addresses enabling technology inroads stemming from major improvements in our understanding of human neural circuitry that promise to enable innovations in the artificial intelligence needed to achieve autonomous system function. Arguments are based on projected military and economic benefits of autonomous systems and extend the historical model established by the CIA\u27s successful UAV program to unconventional warfare (UW) conflicts that the U.S. Air Force finds itself ill-equipped to handle. Counter-arguments are addressed relating to uncontrolled lethal technology, conflict initiation thresholds, and the vulnerability of overreliance on high-technology systems. In making the case for fully automated UAV technology, research provides a strategic future vision for autonomous UAV usage by highlighting the important interaction of artificial intelligence, “smart” wide-area sensors, and cooperative micro UAVs

TOWARDS AUTONOMOUS VERTICAL LANDING ON SHIP-DECKS USING COMPUTER VISION

The objective of this dissertation is to develop and demonstrate autonomous ship-board landing with computer vision. The problem is hard primarily due to the unpredictable stochastic nature of deck motion. The work involves a fundamental understanding of how vision works, what are needed to implement it, how it interacts with aircraft controls, the necessary and sufficient hardware, and software, how it differs from human vision, its limits, and finally the avenues of growth in the context of aircraft landing. The ship-deck motion dataset is provided by the U.S. Navy. This data is analyzed to gain fundamental understanding and is then used to replicate stochastic deck motion in a laboratory setting on a six degrees of freedom motion platform, also called Stewart platform. The method uses a shaping filter derived from the dataset to excite the platform. An autonomous quadrotor UAV aircraft is designed and fabricated for experimental testing of vision-based landing methods. The entire structure, avionics architecture, and flight controls for the aircraft are completely developed in-house. This provides the flexibility and fundamental understanding needed for this research. A fiducial-based vision system is first designed for detection and tracking of ship-deck. This is then utilized to design a tracking controller with the best possible bandwidth to track the deck with minimum error. Systematic experiments are conducted with static, sinusoidal, and stochastic motions to quantify the tracking performance. A feature-based vision system is designed next. Simple experiments are used to quantitatively and qualitatively evaluate the superior robustness of feature-based vision under various degraded visual conditions. This includes: (1) partial occlusion, (2) illumination variation, (3) glare, and (4) water distortion. The weight and power penalty for using feature-based vision are also determined. The results show that it is possible to autonomously land on ship-deck using computer vision alone. An autonomous aircraft can be constructed with only an IMU and a Visual Odometry software running on stereo camera. The aircraft then only needs a monocular, global shutter, high frame rate camera as an extra sensor to detect ship-deck and estimate its relative position. The relative velocity however needs to be derived using Kalman filter on the position signal. For the filter, knowledge of disturbance/motion spectrum is not needed, but a white noise disturbance model is sufficient. For control, a minimum bandwidth of 0.15 Hz is required. For vision, a fiducial is not needed. A feature-rich landing area is all that is required. The limits of the algorithm are set by occlusion(80\% tolerable), illumination (20,000 lux-0.01 lux), angle of landing (up to 45 degrees), 2D nature of features, and motion blur. Future research should extend the capability to 3D features and use of event-based cameras. Feature-based vision is more versatile and human-like than fiducial-based, but at the cost of 20 times higher computing power which is increasingly possible with modern processors. The goal is not an imitation of nature but derive inspiration from it and overcome its limitations. The feature-based landing opens a window towards emulating the best of human training and cognition, without its burden of latency, fatigue, and divided attention

Autonomous High-Precision Landing on a Unmanned Surface Vehicle

THE MAIN GOAL OF THIS THESIS IS THE DEVELOPMENT OF AN AUTONOMOUS HIGH-PRECISION LANDING SYSTEM OF AN UAV IN AN AUTONOMOUS BOATIn this dissertation, a collaborative method for Multi Rotor Vertical Takeoff and Landing (MR-VTOL) Unmanned Aerial Vehicle (UAV)s’ autonomous landing is presented. The majority of common UAV autonomous landing systems adopt an approach in which the UAV scans the landing zone for a predetermined pattern, establishes relative positions, and uses those positions to execute the landing. These techniques have some shortcomings, such as extensive processing being carried out by the UAV itself and requires a lot of computational power. The fact that most of these techniques only work while the UAV is already flying at a low altitude, since the pattern’s elements must be plainly visible to the UAV’s camera, creates an additional issue. An RGB camera that is positioned in the landing zone and pointed up at the sky is the foundation of the methodology described throughout this dissertation. Convolutional Neural Networks and Inverse Kinematics approaches can be used to isolate and analyse the distinctive motion patterns the UAV presents because the sky is a very static and homogeneous environment. Following realtime visual analysis, a terrestrial or maritime robotic system can transmit orders to the UAV. The ultimate result is a model-free technique, or one that is not based on established patterns, that can help the UAV perform its landing manoeuvre. The method is trustworthy enough to be used independently or in conjunction with more established techniques to create a system that is more robust. The object detection neural network approach was able to detect the UAV in 91,57% of the assessed frames with a tracking error under 8%, according to experimental simulation findings derived from a dataset comprising three different films. Also created was a high-level position relative control system that makes use of the idea of an approach zone to the helipad. Every potential three-dimensional point within the zone corresponds to a UAV velocity command with a certain orientation and magnitude. The control system worked flawlessly to conduct the UAV’s landing within 6 cm of the target during testing in a simulated setting.Nesta dissertação, é apresentado um método de colaboração para a aterragem autónoma de Unmanned Aerial Vehicle (UAV)Multi Rotor Vertical Takeoff and Landing (MR-VTOL). A maioria dos sistemas de aterragem autónoma de UAV comuns adopta uma abordagem em que o UAV varre a zona de aterragem em busca de um padrão pré-determinado, estabelece posições relativas, e utiliza essas posições para executar a aterragem. Estas técnicas têm algumas deficiências, tais como o processamento extensivo a ser efectuado pelo próprio UAV e requer muita potência computacional. O facto de a maioria destas técnicas só funcionar enquanto o UAV já está a voar a baixa altitude, uma vez que os elementos do padrão devem ser claramente visíveis para a câmara do UAV, cria um problema adicional. Uma câmara RGB posicionada na zona de aterragem e apontada para o céu é a base da metodologia descrita ao longo desta dissertação. As Redes Neurais Convolucionais e as abordagens da Cinemática Inversa podem ser utilizadas para isolar e analisar os padrões de movimento distintos que o UAV apresenta, porque o céu é um ambiente muito estático e homogéneo. Após análise visual em tempo real, um sistema robótico terrestre ou marítimo pode transmitir ordens para o UAV. O resultado final é uma técnica sem modelo, ou que não se baseia em padrões estabelecidos, que pode ajudar o UAV a realizar a sua manobra de aterragem. O método é suficientemente fiável para ser utilizado independentemente ou em conjunto com técnicas mais estabelecidas para criar um sistema que seja mais robusto. A abordagem da rede neural de detecção de objectos foi capaz de detectar o UAV em 91,57% dos fotogramas avaliados com um erro de rastreio inferior a 8%, de acordo com resultados de simulação experimental derivados de um conjunto de dados composto por três filmes diferentes. Também foi criado um sistema de controlo relativo de posição de alto nível que faz uso da ideia de uma zona de aproximação ao heliporto. Cada ponto tridimensional potencial dentro da zona corresponde a um comando de velocidade do UAV com uma certa orientação e magnitude. O sistema de controlo funcionou sem falhas para conduzir a aterragem do UAV dentro de 6 cm do alvo durante os testes num cenário simulado. Traduzido com a versão gratuita do tradutor - www.DeepL.com/Translato

Remote Sensing methods for power line corridor surveys

AbstractTo secure uninterrupted distribution of electricity, effective monitoring and maintenance of power lines are needed. This literature review article aims to give a wide overview of the possibilities provided by modern remote sensing sensors in power line corridor surveys and to discuss the potential and limitations of different approaches. Monitoring of both power line components and vegetation around them is included. Remotely sensed data sources discussed in the review include synthetic aperture radar (SAR) images, optical satellite and aerial images, thermal images, airborne laser scanner (ALS) data, land-based mobile mapping data, and unmanned aerial vehicle (UAV) data. The review shows that most previous studies have concentrated on the mapping and analysis of network components. In particular, automated extraction of power line conductors has achieved much attention, and promising results have been reported. For example, accuracy levels above 90% have been presented for the extraction of conductors from ALS data or aerial images. However, in many studies datasets have been small and numerical quality analyses have been omitted. Mapping of vegetation near power lines has been a less common research topic than mapping of the components, but several studies have also been carried out in this field, especially using optical aerial and satellite images. Based on the review we conclude that in future research more attention should be given to an integrated use of various data sources to benefit from the various techniques in an optimal way. Knowledge in related fields, such as vegetation monitoring from ALS, SAR and optical image data should be better exploited to develop useful monitoring approaches. Special attention should be given to rapidly developing remote sensing techniques such as UAVs and laser scanning from airborne and land-based platforms. To demonstrate and verify the capabilities of automated monitoring approaches, large tests in various environments and practical monitoring conditions are needed. These should include careful quality analyses and comparisons between different data sources, methods and individual algorithms

Addressing Complexity and Intelligence in Systems Dependability Evaluation

Engineering and computing systems are increasingly complex, intelligent, and open adaptive. When it comes to the dependability evaluation of such systems, there are certain challenges posed by the characteristics of “complexity” and “intelligence”. The first aspect of complexity is the dependability modelling of large systems with many interconnected components and dynamic behaviours such as Priority, Sequencing and Repairs. To address this, the thesis proposes a novel hierarchical solution to dynamic fault tree analysis using Semi-Markov Processes. A second aspect of complexity is the environmental conditions that may impact dependability and their modelling. For instance, weather and logistics can influence maintenance actions and hence dependability of an offshore wind farm. The thesis proposes a semi-Markov-based maintenance model called “Butterfly Maintenance Model (BMM)” to model this complexity and accommodate it in dependability evaluation. A third aspect of complexity is the open nature of system of systems like swarms of drones which makes complete design-time dependability analysis infeasible. To address this aspect, the thesis proposes a dynamic dependability evaluation method using Fault Trees and Markov-Models at runtime.The challenge of “intelligence” arises because Machine Learning (ML) components do not exhibit programmed behaviour; their behaviour is learned from data. However, in traditional dependability analysis, systems are assumed to be programmed or designed. When a system has learned from data, then a distributional shift of operational data from training data may cause ML to behave incorrectly, e.g., misclassify objects. To address this, a new approach called SafeML is developed that uses statistical distance measures for monitoring the performance of ML against such distributional shifts. The thesis develops the proposed models, and evaluates them on case studies, highlighting improvements to the state-of-the-art, limitations and future work

Operational protocols for the use of drones in marine animal research

The use of drones to study marine animals shows promise for the examination of numerous aspects of their ecology, behaviour, health and movement patterns. However, the responses of some marine phyla to the presence of drones varies broadly, as do the general operational protocols used to study them. Inconsistent methodological approaches could lead to difficulties comparing studies and can call into question the repeatability of research. This review draws on current literature and researchers with a wealth of practical experience to outline the idiosyncrasies of studying various marine taxa with drones. We also outline current best practice for drone operation in marine environments based on the literature and our practical experience in the field. The protocols outlined herein will be of use to researchers interested in incorporating drones as a tool into their research on marine animals and will help form consistent approaches for drone-based studies in the future