Real-time Heading Estimation using Perspective Features

There are a large number of commercially available quad-rotor helicopters available from various manufacturers. All of these systems rely on a low cost MEMS based inertial measurement system for stabilization and navigation. These low cost inertial systems are all subject to rapid error growth in their attitude and position estimates unless bounded by external measurements. This thesis created real-time algorithm to integrate measurements from visual cues with measurements from onboard sensors to estimate the attitude position and velocity of a quad-rotor helicopter in a local navigation frame, a system model for the ARDrone, and a feed-back controller for the vehicle\u27s heading. The ARDrone, by Parrot SA, is a low cost quad-rotor helicopter that comes equipped with a variety of sensors including a forward-looking high-definition camera. The vehicle is capable of using its onboard sensors to adequately constrain the errors for pitch and roll in all environments, however the yaw axis is still subject to drift. This work utilizes a RANSAC based vanishing point detection algorithm to provide a reliable heading reference and integrates the vanishing point based heading measurements with the system\u27s onboard heading measurements through an extended Kalman filter. In addition to estimating the drone\u27s heading, the Kalman filter also estimates the position and velocity of the drone as it moves through its environment. This system was able to provide a heading reference with an error of one degree for the drone and was shown to be capable of transitioning between vanishing points when the vehicle needed to change direction. The system also demonstrated that it was capable of generating an estimate of position and velocity. However because the position error was on the order of one meter, the estimate was not accurate enough for autonomous navigation

Composite prototyping and vision based hierarchical control of a quad tilt-wing UAV

As the attention to unmanned systems is increasing, unmanned aerial vehicles (UAVs) are becoming more popular based on the rapid advances in technology and growth in operational experience. The main motivation in this vast research field is to diminish the human driven tasks by employing UAVs in critical civilian and military tasks such as traffic monitoring, disasters, surveillance, reconnaissance and border security. Researchers have been developing featured UAVs with intelligent navigation and control systems on more efficient designs aiming to increase the functionality, flight time and maneuverability. This thesis focuses on the composite prototyping and vision based hierarchical control of a quad tilt-wing aerial vehicle (SUAVI: Sabanci University Unmanned Aerial VehIcle). With the tilt-wing mechanism, SUAVI is one of the most challenging UAV concepts by combining advantages of vertical take-off and landing (VTOL) and horizontal flight. Various composite materials are tested for their mechanical properties and the most suitable one is used for prototyping of the aerial vehicle. A hierarchical control structure which consists of high-level and low-level controllers is developed. A vision based high-level controller generates attitude references for the low-level controllers. A Kalman filter fuses data from low-cost inertial sensors to obtain reliable orientation information. Low-level controllers are typically gravity compensated PID controllers. An image based visual servoing (IBVS) algorithm for VTOL, hovering and trajectory tracking is successfully implemented in simulations. Real flight tests demonstrate satisfactory performance of the developed control algorithms

Vision Based Localization for Multiple UAVs and Mobile Robots

Master'sMASTER OF ENGINEERIN

Optimized Endpoint Delivery Via Unmanned Aerial Vehicles

Unmanned Aerial Vehicles (UAVs) are remotely piloted aircraft with a range of varying applications. Though early adoption of UAVs focused on military applications, surveillance, photography, and agricultural applications are presently on the rise. This work aims to ascertain how UAVs may be employed to elicit deceased transportation times, increased power efficiency, and improved safety. Resulting in optimized end point delivery. A combination of tools and techniques, involving a mathematical model, UAV simulations, redundant control systems, and custom designed electrical and mechanical components were used towards reaching the goal of a 10-kilogram maximum payload delivered 10 miles under 30 minutes. Two UAV prototypes were developed, the second of which (V2) showed promising results. Velocities achieved in V2, in combination with a versatile payload connector and proper networking, allowed for 5-10 mile deliveries of goods less than 8-kilograms to be achieved within a metropolis faster than the 30-minute benchmark

Survey of computer vision algorithms and applications for unmanned aerial vehicles

This paper presents a complete review of computer vision algorithms and vision-based intelligent applications, that are developed in the field of the Unmanned Aerial Vehicles (UAVs) in the latest decade. During this time, the evolution of relevant technologies for UAVs; such as component miniaturization, the increase of computational capabilities, and the evolution of computer vision techniques have allowed an important advance in the development of UAVs technologies and applications. Particularly, computer vision technologies integrated in UAVs allow to develop cutting-edge technologies to cope with aerial perception difficulties; such as visual navigation algorithms, obstacle detection and avoidance and aerial decision-making. All these expert technologies have developed a wide spectrum of application for UAVs, beyond the classic military and defense purposes. Unmanned Aerial Vehicles and Computer Vision are common topics in expert systems, so thanks to the recent advances in perception technologies, modern intelligent applications are developed to enhance autonomous UAV positioning, or automatic algorithms to avoid aerial collisions, among others. Then, the presented survey is based on artificial perception applications that represent important advances in the latest years in the expert system field related to the Unmanned Aerial Vehicles. In this paper, the most significant advances in this field are presented, able to solve fundamental technical limitations; such as visual odometry, obstacle detection, mapping and localization, et cetera. Besides, they have been analyzed based on their capabilities and potential utility. Moreover, the applications and UAVs are divided and categorized according to different criteria.This research is supported by the Spanish Government through the CICYT projects (TRA2015-63708-R and TRA2013-48314-C3-1-R)

NAVIGATION AND AUTONOMOUS CONTROL OF MAVS IN GPS-DENIED ENVIRONMENTS

Ph.DDOCTOR OF PHILOSOPH

Visual control of multi-rotor UAVs

Recent miniaturization of computer hardware, MEMs sensors, and high energy density batteries have enabled highly capable mobile robots to become available at low cost. This has driven the rapid expansion of interest in multi-rotor unmanned aerial vehicles. Another area which has expanded simultaneously is small powerful computers, in the form of smartphones, which nearly always have a camera attached, many of which now contain a OpenCL compatible graphics processing units. By combining the results of those two developments a low-cost multi-rotor UAV can be produced with a low-power onboard computer capable of real-time computer vision. The system should also use general purpose computer vision software to facilitate a variety of experiments. To demonstrate this I have built a quadrotor UAV based on control hardware from the Pixhawk project, and paired it with an ARM based single board computer, similar those in high-end smartphones. The quadrotor weights 980 g and has a flight time of 10 minutes. The onboard computer capable of running a pose estimation algorithm above the 10 Hz requirement for stable visual control of a quadrotor. A feature tracking algorithm was developed for efficient pose estimation, which relaxed the requirement for outlier rejection during matching. Compared with a RANSAC- only algorithm the pose estimates were less variable with a Z-axis standard deviation 0.2 cm compared with 2.4 cm for RANSAC. Processing time per frame was also faster with tracking, with 95 % confidence that tracking would process the frame within 50 ms, while for RANSAC the 95 % confidence time was 73 ms. The onboard computer ran the algorithm with a total system load of less than 25 %. All computer vision software uses the OpenCV library for common computer vision algorithms, fulfilling the requirement for running general purpose software. The tracking algorithm was used to demonstrate the capability of the system by per- forming visual servoing of the quadrotor (after manual takeoff). Response to external perturbations was poor however, requiring manual intervention to avoid crashing. This was due to poor visual controller tuning, and to variations in image acquisition and attitude estimate timing due to using free running image acquisition. The system, and the tracking algorithm, serve as proof of concept that visual control of a quadrotor is possible using small low-power computers and general purpose computer vision software

Vision-Based navigation system for unmanned aerial vehicles

Mención Internacional en el título de doctorThe main objective of this dissertation is to provide Unmanned Aerial Vehicles (UAVs) with a robust navigation system; in order to allow the UAVs to perform complex tasks autonomously and in real-time. The proposed algorithms deal with solving the navigation problem for outdoor as well as indoor environments, mainly based on visual information that is captured by monocular cameras. In addition, this dissertation presents the advantages of using the visual sensors as the main source of data, or complementing other sensors in providing useful information; in order to improve the accuracy and the robustness of the sensing purposes. The dissertation mainly covers several research topics based on computer vision techniques: (I) Pose Estimation, to provide a solution for estimating the 6D pose of the UAV. This algorithm is based on the combination of SIFT detector and FREAK descriptor; which maintains the performance of the feature points matching and decreases the computational time. Thereafter, the pose estimation problem is solved based on the decomposition of the world-to-frame and frame-to-frame homographies. (II) Obstacle Detection and Collision Avoidance, in which, the UAV is able to sense and detect the frontal obstacles that are situated in its path. The detection algorithm mimics the human behaviors for detecting the approaching obstacles; by analyzing the size changes of the detected feature points, combined with the expansion ratios of the convex hull constructed around the detected feature points from consecutive frames. Then, by comparing the area ratio of the obstacle and the position of the UAV, the method decides if the detected obstacle may cause a collision. Finally, the algorithm extracts the collision-free zones around the obstacle, and combining with the tracked waypoints, the UAV performs the avoidance maneuver. (III) Navigation Guidance, which generates the waypoints to determine the flight path based on environment and the situated obstacles. Then provide a strategy to follow the path segments and in an efficient way and perform the flight maneuver smoothly. (IV) Visual Servoing, to offer different control solutions (Fuzzy Logic Control (FLC) and PID), based on the obtained visual information; in order to achieve the flight stability as well as to perform the correct maneuver; to avoid the possible collisions and track the waypoints. All the proposed algorithms have been verified with real flights in both indoor and outdoor environments, taking into consideration the visual conditions; such as illumination and textures. The obtained results have been validated against other systems; such as VICON motion capture system, DGPS in the case of pose estimate algorithm. In addition, the proposed algorithms have been compared with several previous works in the state of the art, and are results proves the improvement in the accuracy and the robustness of the proposed algorithms. Finally, this dissertation concludes that the visual sensors have the advantages of lightweight and low consumption and provide reliable information, which is considered as a powerful tool in the navigation systems to increase the autonomy of the UAVs for real-world applications.El objetivo principal de esta tesis es proporcionar Vehiculos Aereos no Tripulados (UAVs) con un sistema de navegacion robusto, para permitir a los UAVs realizar tareas complejas de forma autonoma y en tiempo real. Los algoritmos propuestos tratan de resolver problemas de la navegacion tanto en ambientes interiores como al aire libre basandose principalmente en la informacion visual captada por las camaras monoculares. Ademas, esta tesis doctoral presenta la ventaja de usar sensores visuales bien como fuente principal de datos o complementando a otros sensores en el suministro de informacion util, con el fin de mejorar la precision y la robustez de los procesos de deteccion. La tesis cubre, principalmente, varios temas de investigacion basados en tecnicas de vision por computador: (I) Estimacion de la Posicion y la Orientacion (Pose), para proporcionar una solucion a la estimacion de la posicion y orientacion en 6D del UAV. Este algoritmo se basa en la combinacion del detector SIFT y el descriptor FREAK, que mantiene el desempeno del a funcion de puntos de coincidencia y disminuye el tiempo computacional. De esta manera, se soluciona el problema de la estimacion de la posicion basandose en la descomposicion de las homografias mundo a imagen e imagen a imagen. (II) Deteccion obstaculos y elusion colisiones, donde el UAV es capaz de percibir y detectar los obstaculos frontales que se encuentran en su camino. El algoritmo de deteccion imita comportamientos humanos para detectar los obstaculos que se acercan, mediante el analisis de la magnitud del cambio de los puntos caracteristicos detectados de referencia, combinado con los ratios de expansion de los contornos convexos construidos alrededor de los puntos caracteristicos detectados en frames consecutivos. A continuacion, comparando la proporcion del area del obstaculo y la posicion del UAV, el metodo decide si el obstaculo detectado puede provocar una colision. Por ultimo, el algoritmo extrae las zonas libres de colision alrededor del obstaculo y combinandolo con los puntos de referencia, elUAV realiza la maniobra de evasion. (III) Guiado de navegacion, que genera los puntos de referencia para determinar la trayectoria de vuelo basada en el entorno y en los obstaculos detectados que encuentra. Proporciona una estrategia para seguir los segmentos del trazado de una manera eficiente y realizar la maniobra de vuelo con suavidad. (IV) Guiado por Vision, para ofrecer soluciones de control diferentes (Control de Logica Fuzzy (FLC) y PID), basados en la informacion visual obtenida con el fin de lograr la estabilidad de vuelo, asi como realizar la maniobra correcta para evitar posibles colisiones y seguir los puntos de referencia. Todos los algoritmos propuestos han sido verificados con vuelos reales en ambientes exteriores e interiores, tomando en consideracion condiciones visuales como la iluminacion y las texturas. Los resultados obtenidos han sido validados con otros sistemas: como el sistema de captura de movimiento VICON y DGPS en el caso del algoritmo de estimacion de la posicion y orientacion. Ademas, los algoritmos propuestos han sido comparados con trabajos anteriores recogidos en el estado del arte con resultados que demuestran una mejora de la precision y la robustez de los algoritmos propuestos. Esta tesis doctoral concluye que los sensores visuales tienen las ventajes de tener un peso ligero y un bajo consumo y, proporcionar informacion fiable, lo cual lo hace una poderosa herramienta en los sistemas de navegacion para aumentar la autonomia de los UAVs en aplicaciones del mundo real.Programa Oficial de Doctorado en Ingeniería Eléctrica, Electrónica y AutomáticaPresidente: Carlo Regazzoni.- Secretario: Fernando García Fernández.- Vocal: Pascual Campoy Cerver