Addressing corner detection issues for machine vision based UAV aerial refueling

The need for developing autonomous aerial refueling capabilities for an Unmanned Aerial Vehicle (UAV) has risen out of the growing importance of UAVs in military and non-military applications. The AAR capabilities would improve the range and the loiter time capabilities of UAVs. A number of AAR techniques have been proposed, based on GPS based measurements and Machine Vision based measurements. The GPS based measurements suffer from distorted data in the wake of the tanker. The MV based techniques proposed the use of optical markers which---when detected---were used to determine relative orientation and position of the tanker and the UAV. The drawback of the MV based techniques is the assumption that all the optical markers are always visible and functional. This research effort proposes an alternative approach where the pose estimation does not depend on optical markers but on Feature Extraction methods. The thesis describes the results of the analysis of specific \u27corner detection\u27 algorithms within a Machine Vision---based approach for the problem of Aerial Refueling for Unmanned Aerial Vehicles. Specifically, the performances of the SUSAN and the Harris corner detection algorithms have been compared. Special emphasis was placed on evaluating their accuracy, the required computational effort, and the robustness of both methods to different sources of noise. Closed loop simulations were performed using a detailed SimulinkRTM -based simulation environment to reproduce docking maneuvers, using the US Air Force refueling boom

Evaluation of machine vision techniques for use within flight control systems

In this thesis, two of the main technical limitations for a massive deployment of Unmanned Aerial Vehicle (UAV) have been considered.;The Aerial Refueling problem is analyzed in the first section. A solution based on the integration of \u27conventional\u27 GPS/INS and Machine Vision sensor is proposed with the purpose of measuring the relative distance between a refueling tanker and UAV. In this effort, comparisons between Point Matching (PM) algorithms and Pose Estimation (PE) algorithms have been developed in order to improve the performance of the Machine Vision sensor. A method of integration based on Extended Kalman Filter (EKF) between GPS/INS and Machine Vision system is also developed with the goal of reducing the tracking error in the \u27pre-contact\u27 to contact and refueling phases.;In the second section of the thesis the issue of Collision Identification (CI) is addressed. A proposed solution consists on the use of Optical Flow (OF) algorithms for the detection of possible collisions in the range of vision of a single camera. The effort includes a study of the performance of different Optical Flow algorithms in different scenarios as well as a method to compute the ideal optical flow with the aim of evaluating the algorithms. An analysis on the suitability for a future real time implementation is also performed for all the analyzed algorithms.;Results of the tests show that the Machine Vision technology can be used to improve the performance in the Aerial Refueling problem. In the Collision Identification problem, the Machine Vision has to be integrated with standard sensors in order to be used inside the Flight Control System

Optical Tracking for Relative Positioning in Automated Aerial Refueling

An algorithm is designed to extract features from video of an air refueling tanker for use in determining the precise relative position of a receiver aircraft. The algorithm is based on receiving a known estimate of the tanker aircraft\u27s position and attitude. The algorithm then uses a known feature model of the tanker to predict the location of those features on a video frame. A corner detector is used to extract features from the video. The measured corners are then associated with known features and tracked from frame to frame. For each frame, the associated features are used to calculate three dimensional pointing vectors to the features of the tanker. These vectors are passed to a navigation algorithm which uses extended Kalman filters, as well as data-linked INS data to solve for the relative position of the tanker. The algorithms were tested using data from a flight test accomplished by the USAF Test Pilot School using a C-12C as a simulated tanker and a Learjet LJ-24 as the simulated receiver. The system was able to provide at least a dozen useful measurements per frame, with and without projection error

Machine vision applications in UAVs for autonomous aerial refueling and runway detection

This research focuses on the application of Machine Vision (MV) techniques and algorithms to the problems of Autonomous Aerial Refueling (AAR) and Runway Detection. In particular, real laboratory based hardware was used in a simulated environment to emulate real-life conditions for AAR. It was shown that the K-Means Clustering Algorithm solution to the Marker Detection problem could be executed at a frame rate of 30 Hz and it averaged a tracking error of less than one pixel while utilizing only 0.16% of the image. It was also shown that the solution to the Runway Detection problem could be executed at a frame rate of 20 Hz which is acceptable for use in an UAV performing reconnaissance work. Data from these tests suggest that both software schemes are suitable for applications in moving vehicles and that the accuracy of the measurements produced by the schemes make them suitable for UAV applications

Guidance, Navigation and Control for UAV Close Formation Flight and Airborne Docking

Unmanned aerial vehicle (UAV) capability is currently limited by the amount of energy that can be stored onboard or the small amount that can be gathered from the environment. This has historically lead to large, expensive vehicles with considerable fuel capacity. Airborne docking, for aerial refueling, is a viable solution that has been proven through decades of implementation with manned aircraft, but had not been successfully tested or demonstrated with UAVs. The prohibitive challenge is the highly accurate and reliable relative positioning performance that is required to dock with a small target, in the air, amidst external disturbances. GNSS-based navigation systems are well suited for reliable absolute positioning, but fall short for accurate relative positioning. Direct, relative sensor measurements are precise, but can be unreliable in dynamic environments. This work proposes an experimentally verified guidance, navigation and control solution that enables a UAV to autonomously rendezvous and dock with a drogue that is being towed by another autonomous UAV. A nonlinear estimation framework uses precise air-to-air visual observations to correct onboard sensor measurements and produce an accurate relative state estimate. The state of the drogue is estimated using known geometric and inertial characteristics and air-to-air observations. Setpoint augmentation algorithms compensate for leader turn dynamics during formation flight, and drogue physical constraints during docking. Vision-aided close formation flight has been demonstrated over extended periods; as close as 4 m; in wind speeds in excess of 25 km/h; and at altitudes as low as 15 m. Docking flight tests achieved numerous airborne connections over multiple flights, including five successful docking manoeuvres in seven minutes of a single flight. To the best of our knowledge, these are the closest formation flights performed outdoors and the first UAV airborne docking

Toward Automated Aerial Refueling: Relative Navigation with Structure from Motion

The USAF\u27s use of UAS has expanded from reconnaissance to hunter/killer missions. As the UAS mission further expands into aerial combat, better performance and larger payloads will have a negative correlation with range and loiter times. Additionally, the Air Force Future Operating Concept calls for \formations of uninhabited refueling aircraft...[that] enable refueling operations partway inside threat areas. However, a lack of accurate relative positioning information prevents the ability to safely maintain close formation flight and contact between a tanker and a UAS. The inclusion of cutting edge vision systems on present refueling platforms may provide the information necessary to support a AAR mission by estimating the position of a trailing aircraft to provide inputs to a UAS controller capable of maintaining a given position. This research examines the ability of SfM to generate relative navigation information. Previous AAR research efforts involved the use of differential GPS, LiDAR, and vision systems. This research aims to leverage current and future imaging technology to compliment these solutions. The algorithm used in this thesis generates a point cloud by determining 3D structure from a sequence of 2D images. The algorithm then utilizes PCA to register the point cloud to a reference model. The algorithm was tested in a real world environment using a 1:7 scale F-15 model. Additionally, this thesis studies common 3D rigid registration algorithms in an effort characterize their performance in the AAR domain. Three algorithms are tested for runtime and registration accuracy with four data sets

Application of Machine Vision in UAVs for Autonomous Target Tracking

This research presents experimental results for the application of Machine Vision (MV) techniques to address the problem of target detection and tracking. The main objective is the design of a prototype UAV surveillance environment to emulate real-life conditions. The model environment for this experiment consists of a target simulated by a small electric train system, located at ground level, and a MV camera mounted on a motion-based apparatus located directly above the model setup. This system is meant to be a non-flying mockup of an aerial robot retrofitted with a MV sensor. Therefore, the final design is a two degree-of-freedom gantry simulating aircraft motions above the ground level at a constant altitude. On the ground level, the design of the landscape is an attempt to achieve a realistic natural landscape within a laboratory setting. Therefore, the scenery consists of small scale trees, bushes, a mountain, and a tunnel system within a 914 mm by 1066 mm boundary. To detect and track the moving train, MV algorithms are implemented in a Matlab/SimulinkRTM based simulation environment. Specifically, image pre-processing techniques and circle detection algorithms are implemented to detect and identify the chimney stack on the train engine. The circle detection algorithms analyzed in this research effort consists of a least squares based method and the Hough transform (HT) method for circle detection. The experimental results will show that the solution to the target detection problem could produce a positive detection rate of 90% during each simulation while utilizing only 56% of the input image. Tracking and timing data also shows that the least squares based target detection method performs substantially better then the HT method. This is evident from the result of using a 1--2 Hz frequency update rate for the SimulinkRTM scheme which is acceptable, in some cases, for use in navigation for a UAV performing scouting and reconnaissance missions. The development of vision-based control strategies, similar to the approach presented in this research, allows UAVs to participate in complex missions involving autonomous target tracking

NDI-based neurocontroller for unmanned combat aerial vehicles during aerial refuelling.

The success of Unmanned Combat Aerial Vehicles (UCAVs) requires further developments in the field of automated aerial refuelling (AAR) and control systems. AAR aircraft models identified thus far do not take the centre of gravity (cg) position movement into account during refuelling. A six-degree-of-freedom aircraft model was combined with a moving cg model for refuelling. The equations of motion for the aircraft in flight refuelling showed the aircraft dynamics to be coupled in the longitudinal and lateral-directional planes when the cg had moved away from the reference point. Applying assumptions specific to the flight conditions, simplified equations of motion were derived. Modal analysis of four cases for the linearised aircraft model during aerial refuelling was conducted. This revealed that the increase in mass was favourable to the stability of the Dutch Roll mode, but the mode did become more oscillatory initially as mass was increased, but as the cg moved forward, the mode became less oscillatory. The opposite was observed with the Phugoid mode. The Short Period Oscillation (SPO) decomposed into two first order modes during refuelling and these remained unchanged during the refuelling process. Three radial basis function (RBF) neural networks (RBFNN) were developed and trained to approximate the inverse plant dynamics and predicted commanded deflections of the elevator, aileron and rudder. Training data required for the network was randomly generated and the desired rates and commanded control surface deflections were computed. The training error was the smallest in the elevator deflection required during refuelling. A basic nonlinear dynamic inversion (NDI) controller without a neural network (NN) was designed for the aircraft. The performance of this controller was not satisfactory. The RBF was combined with the NDI to form a RBFNN-based controller. The longitudinal NDI RBFNN-based controller was less sensitive to modelling errors than the base NDI controller. The lateral NDI RBFNN-based controller’s performance was worse than the longitudinal controller, but showed potential as a technique for future consideration. Including the variation of aircraft inertia in the model has been recommended as further work, as well as exploring other neural network topologies in the NDI NN controller

Integrity Monitoring for Automated Aerial Refueling: A Stereo Vision Approach

Unmanned aerial vehicles (UAVs) increasingly require the capability to y autonomously in close formation including to facilitate automated aerial refueling (AAR). The availability of relative navigation measurements and navigation integrity are essential to autonomous relative navigation. Due to the potential non-availability of the global positioning system (GPS) during military operations, it is highly desirable that relative navigation can be accomplished without the use of GPS. This paper develops two algorithms designed to provide relative navigation measurements solely from a stereo image pair. These algorithms were developed and analyzed in the context of AAR using a stereo camera system modeling that of the KC-46. Algorithms were analyzed in simulation and then in flight test using two C-12C aircraft at the United States Air Force Test Pilot School. The first algorithm, the Vision and Bayesian Inference Based Integrity Monitor (V5), uses Bayesian inference and template matching to return a probability mass function (PMF) describing the position of an observed aircraft. This PMF provides a relative position estimate as well as a protection level--which characterizes the uncertainty of the relative position estimate--thus providing a degree of navigation integrity. Using both simulation and flight test data, mean V5 spherical error was less than one meter and protection levels reliably characterized algorithm uncertainty. The second algorithm, relative pose estimation with computer vision and iterative closest point (R7), uses stereo vision algorithms and the iterative closest point algorithm to return relative position and attitude estimates. Using both simulation and flight test data, mean R7 spherical error was less than 0.5 meters. Additionally, in flight test, mean R7 attitude errors were less than 3° in all axes