Fusion of enhanced and synthetic vision system images for runway and horizon detection

Networked operation of unmanned air vehicles (UAVs) demands fusion of information from disparate sources for accurate flight control. In this investigation, a novel sensor fusion architecture for detecting aircraft runway and horizons as well as enhancing the awareness of surrounding terrain is introduced based on fusion of enhanced vision system (EVS) and synthetic vision system (SVS) images. EVS and SVS image fusion has yet to be implemented in real-world situations due to signal misalignment. We address this through a registration step to align EVS and SVS images. Four fusion rules combining discrete wavelet transform (DWT) sub-bands are formulated, implemented, and evaluated. The resulting procedure is tested on real EVS-SVS image pairs and pairs containing simulated turbulence. Evaluations reveal that runways and horizons can be detected accurately even in poor visibility. Furthermore, it is demonstrated that different aspects of EVS and SVS images can be emphasized by using different DWT fusion rules. The procedure is autonomous throughout landing, irrespective of weather. The fusion architecture developed in this study holds promise for incorporation into manned heads-up displays (HUDs) and UAV remote displays to assist pilots landing aircraft in poor lighting and varying weather. The algorithm also provides a basis for rule selection in other signal fusion applications

Automatic Landing without GPS

Sagem Défense et Sécurité (now Safran Electronics & Defense), a French space and defense company of the SAFRAN group, is working on the next generation of Unmanned Aerial System (UAS). This UAS features a fully automatic Unmanned Aerial Vehicle (UAV) equipped with a state-of-the-art navigation system. This navigation system relies mainly on a high-accuracy Inertial Measurement Unit (IMU) coupled with a GPS receiver. But the GPS is known to be easy to jam, either naturally (solar flare for example) or intentionally. In the event of a loss of GPS signal, the navigation system is not able anymore to provide accurate position and speed information to the Flight Controller (FC). Deprived of reliable position and speed information the FC is not able to guide the UAV safely to the ground. So the goal of the project detailed in this report is to add to the existing UAS the ability to land safely in case of a GPS loss. At the core of the solution described in this report is a sensor fusion algorithm taking as input inertial, vision based, barometric, laser and azimuthal measurements. The filter is using all these measurements to establish reliable position and speed estimates. Even if very reliable systems enabling automatic landing without GPS exist today; they all require heavy and expensive ground equipment. This is why SAGEM decided to develop its own solution using more embedded sensors and less ground equipment. This is a first step toward a fully embedded automatic landing system nondependent on GPS availability, a very active field of research today. All the tests done during the thesis and presented in this report shows the efficiency and robustness of this solution

UAV tracking module proposal based on a regulative comparison between manned and unmanned aviation

Purpose: The aim of this study is twofold. First is to compare manned and unmanned aviation regulations in the context of ICAO Annexes to identify potential deficiencies in the international UAV legislations. Second is to propose a UAV monitoring module work flow as a solution to identified deficiencies in the international UAV regulations. Design/methodology: In the present study, firstly the regulations used in manned aviation were summarized in the context of ICAO Annexes. Then along with an overview of the use of UAVs, international UAV regulations have been reviewed with a general perspective. In addition, a comparison was made on whether contents of ICAO Annexes find a place in common international UAV regulations in order to understand areas to be developed in the international UAV regulations, and to better understand the different principles between manned and unmanned air transport. In the last section, we present a UAV tracking module (UAVTram) in line with the above-mentioned comparison between manned and unmanned aviation and the identified deficiencies in the international UAV regulations. Findings: The international UAV regulations should be developed on the basis of airport airspace use, detection, liabilities, sanctions of violations, and updating of regulation. Proposed UAVTram has potential to offer real-time tracking and detection of UAVs as a solution to malicious use of UAVs. Research limitations/implications: Our study is not exempt from limitations. Firstly, we didn’t review all UAV regulations because it needs a considerable amount of efforts to check out all the UAV regulations pertinent to different areas of the world. It is the same case for manned aviation as we used only ICAO Annexes to contextually compare with UAV regulations. Practical implications: From the practical perspective, studies introducing new technologies such as systems that help detection of remote pilots causing trouble and agile defense systems will give valuable insights to remove individual UAV threats. Originality/value: We didn’t find any study aiming to compare manned and unmanned aviation rules in search of finding potential deficiencies in the UAV regulations. Our study adopts such an approach. Moreover, our solution proposal here uses Bluetooth 5.0 technology mounted on stationary transmitters which provides more effective range with higher data transfer. Another advantage is that this work is projected to be supported by Turkish civil aviation authority, DGCA. This may accelerate efforts to make required real-time tests.Peer Reviewe

Using learning from demonstration to enable automated flight control comparable with experienced human pilots

Modern autopilots fall under the domain of Control Theory which utilizes Proportional Integral Derivative (PID) controllers that can provide relatively simple autonomous control of an aircraft such as maintaining a certain trajectory. However, PID controllers cannot cope with uncertainties due to their non-adaptive nature. In addition, modern autopilots of airliners contributed to several air catastrophes due to their robustness issues. Therefore, the aviation industry is seeking solutions that would enhance safety. A potential solution to achieve this is to develop intelligent autopilots that can learn how to pilot aircraft in a manner comparable with experienced human pilots. This work proposes the Intelligent Autopilot System (IAS) which provides a comprehensive level of autonomy and intelligent control to the aviation industry. The IAS learns piloting skills by observing experienced teachers while they provide demonstrations in simulation. A robust Learning from Demonstration approach is proposed which uses human pilots to demonstrate the task to be learned in a flight simulator while training datasets are captured. The datasets are then used by Artificial Neural Networks (ANNs) to generate control models automatically. The control models imitate the skills of the experienced pilots when performing the different piloting tasks while handling flight uncertainties such as severe weather conditions and emergency situations. Experiments show that the IAS performs learned skills and tasks with high accuracy even after being presented with limited examples which are suitable for the proposed approach that relies on many single-hidden-layer ANNs instead of one or few large deep ANNs which produce a black-box that cannot be explained to the aviation regulators. The results demonstrate that the IAS is capable of imitating low-level sub-cognitive skills such as rapid and continuous stabilization attempts in stormy weather conditions, and high-level strategic skills such as the sequence of sub-tasks necessary to takeoff, land, and handle emergencies

Vision Aided Automatic Landing System for Fixed Wing UAV

Abstract-In this paper, we present a multi-sensor system for automatic landing of fixed wing UAVs. The system is composed of a high precision aircraft controller and a vision module which is currently used for detection and tracking of runways. Designing the system we paid special attention to its robustness. The runway detection algorithm uses a maximum amount of information in images and works with high level geometrical models. It allows detecting a runway under different weather conditions even if only a small part is visible in the image. In order to increase landing reliability under sub-optimal wind conditions, an additional loop was introduced into the altitude controller. All control and image processing is performed onboard. The system has been successfully tested in flight experiments with two different fixed wing platforms at various weather conditions, in summer, fall and winter

Optical Navigation Sensor for Runway Relative Positioning of Aircraft during Final Approach

Precise navigation is often performed by sensor fusion of different sensors. Among these sensors, optical sensors use image features to obtain the position and attitude of the camera. Runway relative navigation during final approach is a special case where robust and continuous detection of the runway is required. This paper presents a robust threshold marker detection method for monocular cameras and introduces an on-board real-time implementation with flight test results. Results with narrow and wide field-of-view optics are compared. The image processing approach is also evaluated on image data captured by a different on-board system. The pure optical approach of this paper increases sensor redundancy because it does not require input from an inertial sensor as most of the robust runway detectors

Helicopter Autonomous Ship Landing System

This research focuses on developing a helicopter autonomous ship landing algorithm based on the real helicopter ship landing procedure which is already proven and currently used by Navy pilots. It encompasses the entire ship landing procedure from approach to landing using a pilot-inspired vision-based navigation system. The present thesis focuses on the first step towards achieving this overarching objective, which involves modeling the flight dynamics and control of a helicopter and some preliminary simulations of a UH-60 (Blackhawk) helicopter landing on a ship. The airframe of the helicopter is modeled as a rigid body along with rotating articulated blades that can undergo flap, lag and pitch motions about the root. A UH-60 helicopter is used for a representative model due to its ample simulation and flight test data. Modeling a UH-60 helicopter is based on Blade Element Momentum Theory (BEMT), rotor aerodynamics with the Pitt-Peters linear inflow model, empennage aerodynamics and rigid body dynamics for fuselage. For the blade dynamics, the cyclic (1/rev) and collective pitch motions are prescribed and the blade (1/rev) flap and lag motions are obtained as a response to the aerodynamic and inertial forces. The helicopter control inputs and translational and attitude dynamics obtained from the model are validated with flight test data at various speeds and attitude. A linearized model is extracted based on a first-order Taylor series expansion of the nonlinear system about an equilibrium point for the purpose of determining the stability of the dynamic system, investigating sensitivity to gusts, and designing a model-based flight control system. Combined vision-based navigation and Linear Quadratic Regulator (LQR) for set-point tracking is used for disturbance rejection and tracking states. A rotatable camera is used for identifying the relative position of the helicopter with respect to the ship. Based on the position, a corresponding trajectory is computed. Considering the trade-off between transient responses and control efforts, gains for the LQR controller are chosen carefully and realistically. A fully autonomous flight is simulated from approach to landing on a ship. It consists of initial descent, steady forward flight, steady coordinated turn, deceleration, and final landing. Corresponding to each maneuver, relevant linearized model is used and gains are tuned. By using X-plane flight simulator program, the simulation data which include fuselage attitude and position at each time step are visualized with a single flight deck ship. This method allows an aircraft to land on a ship autonomously while maintaining high level of safety and accuracy without the need to capture the ship deck motions, however, by using a camera, and any other additional sensors, which will provide the accurate location of the ship relative to the helicopter. This method is not only relevant for a particular helicopter, but for any types of VTOL aircraft, manned or unmanned. Hence, it can improve the level of safety by preventing human errors that may occur during landing on a ship

Small unmanned airborne systems to support oil and gas pipeline monitoring and mapping

Acknowledgments We thank Johan Havelaar, Aeryon Labs Inc., AeronVironment Inc. and Aeronautics Inc. for kindly permitting the use of materials in Fig. 1.Peer reviewedPublisher PD

Helicopter Autonomous Ship Landing System

This research focuses on developing a helicopter autonomous ship landing algorithm based on the real helicopter ship landing procedure which is already proven and currently used by Navy pilots. It encompasses the entire ship landing procedure from approach to landing using a pilot-inspired vision-based navigation system. The present thesis focuses on the first step towards achieving this overarching objective, which involves modeling the flight dynamics and control of a helicopter and some preliminary simulations of a UH-60 (Blackhawk) helicopter landing on a ship. The airframe of the helicopter is modeled as a rigid body along with rotating articulated blades that can undergo flap, lag and pitch motions about the root. A UH-60 helicopter is used for a representative model due to its ample simulation and flight test data. Modeling a UH-60 helicopter is based on Blade Element Momentum Theory (BEMT), rotor aerodynamics with the Pitt-Peters linear inflow model, empennage aerodynamics and rigid body dynamics for fuselage. For the blade dynamics, the cyclic (1/rev) and collective pitch motions are prescribed and the blade (1/rev) flap and lag motions are obtained as a response to the aerodynamic and inertial forces. The helicopter control inputs and translational and attitude dynamics obtained from the model are validated with flight test data at various speeds and attitude. A linearized model is extracted based on a first-order Taylor series expansion of the nonlinear system about an equilibrium point for the purpose of determining the stability of the dynamic system, investigating sensitivity to gusts, and designing a model-based flight control system. Combined vision-based navigation and Linear Quadratic Regulator (LQR) for set-point tracking is used for disturbance rejection and tracking states. A rotatable camera is used for identifying the relative position of the helicopter with respect to the ship. Based on the position, a corresponding trajectory is computed. Considering the trade-off between transient responses and control efforts, gains for the LQR controller are chosen carefully and realistically. A fully autonomous flight is simulated from approach to landing on a ship. It consists of initial descent, steady forward flight, steady coordinated turn, deceleration, and final landing. Corresponding to each maneuver, relevant linearized model is used and gains are tuned. By using X-plane flight simulator program, the simulation data which include fuselage attitude and position at each time step are visualized with a single flight deck ship. This method allows an aircraft to land on a ship autonomously while maintaining high level of safety and accuracy without the need to capture the ship deck motions, however, by using a camera, and any other additional sensors, which will provide the accurate location of the ship relative to the helicopter. This method is not only relevant for a particular helicopter, but for any types of VTOL aircraft, manned or unmanned. Hence, it can improve the level of safety by preventing human errors that may occur during landing on a ship

Helicopter Autonomous Ship Landing System

This research focuses on developing a helicopter autonomous ship landing algorithm based on the real helicopter ship landing procedure which is already proven and currently used by Navy pilots. It encompasses the entire ship landing procedure from approach to landing using a pilot-inspired vision-based navigation system. The present thesis focuses on the first step towards achieving this overarching objective, which involves modeling the flight dynamics and control of a helicopter and some preliminary simulations of a UH-60 (Blackhawk) helicopter landing on a ship. The airframe of the helicopter is modeled as a rigid body along with rotating articulated blades that can undergo flap, lag and pitch motions about the root. A UH-60 helicopter is used for a representative model due to its ample simulation and flight test data. Modeling a UH-60 helicopter is based on Blade Element Momentum Theory (BEMT), rotor aerodynamics with the Pitt-Peters linear inflow model, empennage aerodynamics and rigid body dynamics for fuselage. For the blade dynamics, the cyclic (1/rev) and collective pitch motions are prescribed and the blade (1/rev) flap and lag motions are obtained as a response to the aerodynamic and inertial forces. The helicopter control inputs and translational and attitude dynamics obtained from the model are validated with flight test data at various speeds and attitude. A linearized model is extracted based on a first-order Taylor series expansion of the nonlinear system about an equilibrium point for the purpose of determining the stability of the dynamic system, investigating sensitivity to gusts, and designing a model-based flight control system. Combined vision-based navigation and Linear Quadratic Regulator (LQR) for set-point tracking is used for disturbance rejection and tracking states. A rotatable camera is used for identifying the relative position of the helicopter with respect to the ship. Based on the position, a corresponding trajectory is computed. Considering the trade-off between transient responses and control efforts, gains for the LQR controller are chosen carefully and realistically. A fully autonomous flight is simulated from approach to landing on a ship. It consists of initial descent, steady forward flight, steady coordinated turn, deceleration, and final landing. Corresponding to each maneuver, relevant linearized model is used and gains are tuned. By using X-plane flight simulator program, the simulation data which include fuselage attitude and position at each time step are visualized with a single flight deck ship. This method allows an aircraft to land on a ship autonomously while maintaining high level of safety and accuracy without the need to capture the ship deck motions, however, by using a camera, and any other additional sensors, which will provide the accurate location of the ship relative to the helicopter. This method is not only relevant for a particular helicopter, but for any types of VTOL aircraft, manned or unmanned. Hence, it can improve the level of safety by preventing human errors that may occur during landing on a ship