Development and Deployment of a Dynamic Soaring Capable UAV using Reinforcement Learning

Dynamic soaring (DS) is a bio-inspired flight maneuver in which energy can be gained by flying through regions of vertical wind gradient such as the wind shear layer. With reinforcement learning (RL), a fixed wing unmanned aerial vehicle (UAV) can be trained to perform DS maneuvers optimally for a variety of wind shear conditions. To accomplish this task, a 6-degreesof- freedom (6DoF) flight simulation environment in MATLAB and Simulink has been developed which is based upon an off-the-shelf unmanned aerobatic glider. A combination of high-fidelity Reynolds-Averaged Navier-Stokes (RANS) computational fluid dynamics (CFD) in ANSYS Fluent and low-fidelity vortex lattice (VLM) method in Surfaces was employed to build a complete aerodynamic model of the UAV. Deep deterministic policy gradient (DDPG), an actor-critic RL algorithm, was used to train a closed-loop Path Following (PF) agent and an Unguided Energy- Seeking (UES) agent. Several generations of the PF agent were presented, with the final generation capable of controlling the climb and turn rate of the UAV to follow a closed-loop waypoint path with variable altitude. This must be paired with a waypoint optimizing agent to perform loitering DS. The UES agent was designed to perform traveling DS in a fixed wind shear condition. It was proven to extract energy from the wind shear to extend flight time during training but did not accomplish sustainable dynamic soaring. Further RL training is required for both agents. Recommendations on how to deploy an RL agent on a physical UAV are discussed

Collaborative UAV Surveillance

Autonomous collaborative robotics is a topic of significant interest to groups such as the Air Force Research Lab (AFRL) and the National Aeronautics and Space Administration (NASA). These two groups have been developing systems for the operation of autonomous vehicles over the past several years, but each system has several critical drawbacks. AFRL’s Unmanned Systems Autonomy Services (UxAS) supports pathfinding for multiple tasks performed by groups of vehicles, but has no formal verification, very little physical flight time, and no concept of collision avoidance. NASA’s Independent Configurable Architecture for Reliable Operations of Unmanned Systems (ICAROUS) has collision avoidance, partial formal verification, and thousands of hours of physical flight time, but has no concept of collaboration. AFRL and NASA each wanted to incorporate the features of the other’s software into their own, and so the CRoss-Application Translator for Operational Unmanned Systems (CRATOUS) was created. CRATOUS creates a communication bridge between UxAS and ICAROUS, allowing for full feature integration of the two system. This combined software is the first system that allows for the safe and reliable cooperation of groups of unmanned vehicles

Optimal UAV Path Planning for Tracking a Moving Ground Vehicle with a Gimbaled Camera

This research develops a path planning algorithm that autonomously controls a UAV to provide convoy overwatch. The optimization algorithm determines the best path to y through developing a cost function that minimizes the control effort of the UAV and the deviation from a desired slant range. A heuristic-based algorithm was developed and implemented on the autopilot to approximate the optimal solution. In flight test, the UAV successfully tracked a moving ground vehicle by continually placing the UAV\u27s loiter point directly above the ground vehicle\u27s current location. This method was called the \follow-me mode and provided the baseline for real-world UAV convoy overwatch. The follow-me mode resulted in a cost function value that was 113 times greater than the optimal path. Through an in-depth analysis, the heuristic-based approach reduced this ratio down to only 7.5 times greater than the optimal path. The data collected shows tremendous promise for improving autonomous UAV performance through optimal control techniques

Collision Avoidance and Navigation of UAS Using Vision-Based Proportional Navigation

Electro-optical devices have received considerable interest due to their light weight, low cost, and low algorithm requirements with respect to computational power. In this thesis, vision-based guidance laws are developed to provide sense and avoid capabilities for unmanned aerial vehicles (UAVs) operating in complex environments with multiple static and dynamic collision threats. These collision avoidance guidance laws are based on the principle of proportional navigation (Pro-Nav), which states that a UAV is on a collision course with another vehicle or object if the line-of-sight (LOS) angles to the object remain constant. The guidance laws are designed for use with monocular electro-optical devices, which provide information on the LOS angles to potential collision threats, but not the range. The development of these guidance laws propagates from an investigation into numerous methods of Pro-Nav based guidance, including the use of LOS rate thresholding, avoidance of the most imminent threat detected, and objective-based cost optimization. The collision avoidance guidance laws were applied to nonlinear, six degree-of-freedom UAV models in various simulation environments including a varying number of static and dynamic obstacles. A final form of the avoidance law, determined from these simulation studies, was applied to a small-scale UAV model flying through a virtual urban environment, which utilizes camera-in-the-loop simulation techniques. The final results of these studies showed that the most effective approach was to implement a cost function-based avoidance law that includes a term based on the Pro-Nav intercept heading for a desired waypoint and avoidance terms for all obstacles in view that pose a collision threat. Obstacle avoidance headings in the cost function are based on the difference in the obstacle LOS rates from the magnitude of the minimum safe LOS rate. When applied to UAV simulations in a virtual urban environment, this guidance law provided successful avoidance for the case of a single building, maintained a safe heading through an urban canyon, and determined the safest path through a complex urban layout. For the case of the complex urban layout, a single collision during flight occurred due to a lack of visual feature points to contribute to the avoidance law calculation

ASSESSMENT OF ELECTRO-OPTICAL IMAGING TECHNOLOGY FOR UNMANNED AERIAL SYSTEM NAVIGATION IN A GPS-DENIED ENVIRONMENT

Navigation systems of unmanned aircraft systems (UAS) are heavily dependent on the availability of Global Positioning Systems (GPS) or other Global Navigation Satellite Systems (GNSS). Although inertial navigation systems (INS) can provide position and velocity of an aircraft based on acceleration measurements, the information degrades over time and reduces the capability of the system. In a GPS-denied environment, a UAS must utilize alternative sensor sources for navigating. This thesis presents preliminary evaluation results on the usage of onboard down-looking electro-optical sensors and image matching techniques to assist in GPS-free navigation of aerial platforms. Following the presentation of the fundamental mathematics behind the proposed concept, the thesis analyzes the key results from three flight campaign experiments that use different sets of sensors to collect data. Each of the flight experiments explores different sensor setups, assesses a variety of image processing methods, looks at different terrain environments, and reveals limitations related to the proposed approach. In addition, an attempt to incorporate navigational aid solutions into a navigation system using a Kalman filter is demonstrated. The thesis concludes with recommendations for future research on developing an integrated navigation system that relies on inertial measurement unit data complemented by the positional fixes from the image-matching technique.Outstanding ThesisCivilian, DSO National Laboratories, SingaporeApproved for public release. Distribution is unlimited

UAV Parameter Estimation with Gaussian Process Approximations

Unmanned Aerial Vehicles (UAVs) provide an alternative to manned aircraft for risk associated missions and applications where sizing constraints require miniaturized flying platforms. UAVs are currently utilised in an array of applications ranging from civilian research to military battlegrounds. A part of the development process for UAVs includes constructing a flight model. This model can be used for modern flight controller design and to develop high fidelity flight simulators. Furthermore, it also has a role in analysing stability, control and handling qualities of the platform. Developing such a model involves estimating stability and control parameters from flight data. These map the platform's control inputs to its dynamic response. The modeling process is labor intensive and requires coarse approximations. Similarly, models constructed through flight tests are only applicable to a narrow flight envelope and classical system identification approaches require prior knowledge of the model structure, which, in some instances may only be partially known. This thesis attempts to find a solution to these problems by introducing a new system identification method based on dependent Gaussian processes. The new method would allow for high fidelity non-linear flight dynamic models to be constructed through experimental data. The work is divided into two main components. The first part entails the development of an algorithm that captures cross coupling between input parameters, and learns the system stability and control derivatives. The algorithm also captures any dependencies embodied in the outputs. The second part focuses on reducing the heavy computational cost, which is a deterrent to learning the model from large test flight data sets. In addition, it explores the capabilities of the model to capture any non-stationary behavior in the aerodynamic coefficients. A modeling technique was developed that uses an additive sparse model to combine global and local Gaussian processes to learn a multi-output system. Having a combined approximation makes the model suitable for all regions of the flight envelope. In an attempt to capture the global properties, a new sampling method is introduced to gather information about the output correlations. Local properties were captured using a non-stationary covariance function with KD-trees for neighbourhood selection. This makes the model scalable to learn from high dimensional large-scale data sets. The thesis provides both theoretical underpinnings and practical applications of this approach. The theory was tested in simulation on a highly coupled oblique wing aircraft and was demonstrated on a delta-wing UAV platform using real flight data. The results were compared against an alternative parametric model and demonstrated robustness, improved identification of coupling between flight modes, sound ability to provide uncertainty estimates, and potential to be applied to a broader flight envelope

UAV Parameter Estimation with Gaussian Process Approximations

Unmanned Aerial Vehicles (UAVs) provide an alternative to manned aircraft for risk associated missions and applications where sizing constraints require miniaturized flying platforms. UAVs are currently utilised in an array of applications ranging from civilian research to military battlegrounds. A part of the development process for UAVs includes constructing a flight model. This model can be used for modern flight controller design and to develop high fidelity flight simulators. Furthermore, it also has a role in analysing stability, control and handling qualities of the platform. Developing such a model involves estimating stability and control parameters from flight data. These map the platform's control inputs to its dynamic response. The modeling process is labor intensive and requires coarse approximations. Similarly, models constructed through flight tests are only applicable to a narrow flight envelope and classical system identification approaches require prior knowledge of the model structure, which, in some instances may only be partially known. This thesis attempts to find a solution to these problems by introducing a new system identification method based on dependent Gaussian processes. The new method would allow for high fidelity non-linear flight dynamic models to be constructed through experimental data. The work is divided into two main components. The first part entails the development of an algorithm that captures cross coupling between input parameters, and learns the system stability and control derivatives. The algorithm also captures any dependencies embodied in the outputs. The second part focuses on reducing the heavy computational cost, which is a deterrent to learning the model from large test flight data sets. In addition, it explores the capabilities of the model to capture any non-stationary behavior in the aerodynamic coefficients. A modeling technique was developed that uses an additive sparse model to combine global and local Gaussian processes to learn a multi-output system. Having a combined approximation makes the model suitable for all regions of the flight envelope. In an attempt to capture the global properties, a new sampling method is introduced to gather information about the output correlations. Local properties were captured using a non-stationary covariance function with KD-trees for neighbourhood selection. This makes the model scalable to learn from high dimensional large-scale data sets. The thesis provides both theoretical underpinnings and practical applications of this approach. The theory was tested in simulation on a highly coupled oblique wing aircraft and was demonstrated on a delta-wing UAV platform using real flight data. The results were compared against an alternative parametric model and demonstrated robustness, improved identification of coupling between flight modes, sound ability to provide uncertainty estimates, and potential to be applied to a broader flight envelope