Aerial Object Following Using Visual Fuzzy Servoing

This article presents a visual servoing system to follow a 3D moving object by a Micro Unmanned Aerial Vehicle (MUAV). The presented control strategy is based only on the visual information given by an adaptive tracking method based on the colour information. A visual fuzzy system has been developed for servoing the camera situated on a rotary wing MAUV, that also considers its own dynamics. This system is focused on continuously following of an aerial moving target object, maintaining it with a fixed safe distance and centred on the image plane. The algorithm is validated on real flights on outdoors scenarios, showing the robustness of the proposed systems against winds perturbations, illumination and weather changes among others. The obtained results indicate that the proposed algorithms is suitable for complex controls task, such object following and pursuit, flying in formation, as well as their use for indoor navigatio

Heterogeneous parallelization for object detection and tracking in UAVs.

Recent technical advancements in both fields of unmanned aerial vehicles (UAV) control and artificial intelligence (AI) have made a certain realm of applications possible. However, one of the main problems in integration of these two areas is the bottle-neck of computing AI applications on UAV's resource limited platform. One of the main solution for this problem is that AI and control software from one side and computing hardware mounted on UAV from the other side be adopted together based on the main constraints of the resource limited computing platform on UAV. Basically, the target constraints of such adaptation are performance, energy efficiency, and accuracy. In this paper, we propose a strategy to integrate and adopt the commonly used object detection and tracking algorithm and UAV control software to be executed on a heterogeneous resource limited computing units on a UAV. For object detection, a convolutional neural network (CNN) algorithm is used. For object tracking, a novel algorithm is proposed that can execute along with object tracking via sequential stream data. For UAV control, a Gain-Scheduled PID controller is designed that steers the UAV by continuously manipulation of the actuators based on the stream data from the tracking unit and dynamics of the UAV. All the algorithms are adopted to be executed on a heterogeneous platform including NVIDIA Jetson TX2 embedded computer and an ARM Cortex M4. The observation from real-time operation of the platform shows that using the proposed platform reduces the power consumption by 53.69% in contrast with other existing methods while having marginal penalty for object detection and tracking parts

Development of Unmanned Aerial Vehicle (Quadcopter)With Real-Time Object Tracking

In the previous decade, Unmanned Aerial Vehicles (UAVs) have turned into a subject of enthusiasm for some exploration associations. UAVs are discovering applications in different regions going from military applications to activity reconnaissance. This thesis is an overview of a particular sort of UAV called quadrotor or quadcopter. Scientists are often picking quadrotors for their exploration because a quadrotor can precisely and productively perform assignments that future of high hazard for a human pilot to perform. This thesis includes the dynamic models of a quadrotor and model-autonomous control systems. It also explains the complete description of developed quadcopter used for surveillance purpose with real-time object detection. In the present time, the focus has moved to outlining autonomous quadrotors. Ultimately, it examines the potential applications of quadrotors and their part in multi-operators frameworks. The Unmanned aerial vehicle (Quadcopter) has been developed that could be used for search and surveillance purpose. This project comprised of both hardware and software part. The hardware part comprised of the development of unmanned aerial vehicle (Quadcopter). The main components that were used in this project are KK2 flight controller board, outrunner brushless DC motor, Electronic Speed Controllers (ESC), GPS (Global Positioning System) receiver, video transmitter and receiver, HD (High Definition) camera, RC (Radio Controlled) transmitter and receiver. Software part comprised of real-time object detection and tracking algorithm for detecting and tracking of human beings that were done with the help of Matlab software. After achieving the stable flight, the camera installed on the quadcopter would transmit a video signal to the receiver placed on the ground station. Video signal from the receiver would then be transferred to Matlab software for further processing or for tracking human beings using real-time object detection and tracking algorith

Convolutional neural network-based real-time object detection and tracking for parrot AR drone 2.

Recent advancements in the field of Artificial Intelligence (AI) have provided an opportunity to create autonomous devices, robots, and machines characterized particularly with the ability to make decisions and perform tasks without human mediation. One of these devices, Unmanned Aerial Vehicles (UAVs) or drones are widely used to perform tasks like surveillance, search and rescue, object detection and target tracking, parcel delivery (recently started by Amazon), and many more. The sensitivity in performing said tasks demands that drones must be efficient and reliable. For this, in this paper, an approach to detect and track the target object, moving or still, for a drone is presented. The Parrot AR Drone 2 is used for this application. Convolutional Neural Network (CNN) is used for object detection and target tracking. The object detection results show that CNN detects and classifies object with a high level of accuracy (98%). For real-time tracking, the tracking algorithm responds faster than conventionally used approaches, efficiently tracking the detected object without losing it from sight. The calculations based on several iterations exhibit that the efficiency achieved for target tracking is 96.5%

Heterogeneous Parallelization for Object Detection and Tracking in UAVs

Recent technical advancements in both fields of unmanned aerial vehicles (UAV) control and artificial intelligence (AI) have made a certain realm of applications possible. However, one of the main problems in integration of these two areas is the bottle-neck of computing AI applications on UAV & x2019;s resource limited platform. One of the main solution for this problem is that AI and control software from one side and computing hardware mounted on UAV from the other side be adopted together based on the main constraints of the resource limited computing platform on UAV. Basically, the target constraints of such adaptation are performance, energy efficiency, and accuracy. In this paper, we propose a strategy to integrate and adopt the commonly used object detection and tracking algorithm and UAV control software to be executed on a heterogeneous resource limited computing units on a UAV. For object detection, a convolutional neural network (CNN) algorithm is used. For object tracking, a novel algorithm is proposed that can execute along with object tracking via sequential stream data. For UAV control, a Gain-Scheduled PID controller is designed that steers the UAV by continuously manipulation of the actuators based on the stream data from the tracking unit and dynamics of the UAV. All the algorithms are adopted to be executed on a heterogeneous platform including NVIDIA Jetson TX2 embedded computer and an ARM Cortex M4. The observation from real-time operation of the platform shows that using the proposed platform reduces the power consumption by 53.69 & x0025; in contrast with other existing methods while having marginal penalty for object detection and tracking parts

Vision-Based navigation system for unmanned aerial vehicles

Mención Internacional en el título de doctorThe main objective of this dissertation is to provide Unmanned Aerial Vehicles (UAVs) with a robust navigation system; in order to allow the UAVs to perform complex tasks autonomously and in real-time. The proposed algorithms deal with solving the navigation problem for outdoor as well as indoor environments, mainly based on visual information that is captured by monocular cameras. In addition, this dissertation presents the advantages of using the visual sensors as the main source of data, or complementing other sensors in providing useful information; in order to improve the accuracy and the robustness of the sensing purposes. The dissertation mainly covers several research topics based on computer vision techniques: (I) Pose Estimation, to provide a solution for estimating the 6D pose of the UAV. This algorithm is based on the combination of SIFT detector and FREAK descriptor; which maintains the performance of the feature points matching and decreases the computational time. Thereafter, the pose estimation problem is solved based on the decomposition of the world-to-frame and frame-to-frame homographies. (II) Obstacle Detection and Collision Avoidance, in which, the UAV is able to sense and detect the frontal obstacles that are situated in its path. The detection algorithm mimics the human behaviors for detecting the approaching obstacles; by analyzing the size changes of the detected feature points, combined with the expansion ratios of the convex hull constructed around the detected feature points from consecutive frames. Then, by comparing the area ratio of the obstacle and the position of the UAV, the method decides if the detected obstacle may cause a collision. Finally, the algorithm extracts the collision-free zones around the obstacle, and combining with the tracked waypoints, the UAV performs the avoidance maneuver. (III) Navigation Guidance, which generates the waypoints to determine the flight path based on environment and the situated obstacles. Then provide a strategy to follow the path segments and in an efficient way and perform the flight maneuver smoothly. (IV) Visual Servoing, to offer different control solutions (Fuzzy Logic Control (FLC) and PID), based on the obtained visual information; in order to achieve the flight stability as well as to perform the correct maneuver; to avoid the possible collisions and track the waypoints. All the proposed algorithms have been verified with real flights in both indoor and outdoor environments, taking into consideration the visual conditions; such as illumination and textures. The obtained results have been validated against other systems; such as VICON motion capture system, DGPS in the case of pose estimate algorithm. In addition, the proposed algorithms have been compared with several previous works in the state of the art, and are results proves the improvement in the accuracy and the robustness of the proposed algorithms. Finally, this dissertation concludes that the visual sensors have the advantages of lightweight and low consumption and provide reliable information, which is considered as a powerful tool in the navigation systems to increase the autonomy of the UAVs for real-world applications.El objetivo principal de esta tesis es proporcionar Vehiculos Aereos no Tripulados (UAVs) con un sistema de navegacion robusto, para permitir a los UAVs realizar tareas complejas de forma autonoma y en tiempo real. Los algoritmos propuestos tratan de resolver problemas de la navegacion tanto en ambientes interiores como al aire libre basandose principalmente en la informacion visual captada por las camaras monoculares. Ademas, esta tesis doctoral presenta la ventaja de usar sensores visuales bien como fuente principal de datos o complementando a otros sensores en el suministro de informacion util, con el fin de mejorar la precision y la robustez de los procesos de deteccion. La tesis cubre, principalmente, varios temas de investigacion basados en tecnicas de vision por computador: (I) Estimacion de la Posicion y la Orientacion (Pose), para proporcionar una solucion a la estimacion de la posicion y orientacion en 6D del UAV. Este algoritmo se basa en la combinacion del detector SIFT y el descriptor FREAK, que mantiene el desempeno del a funcion de puntos de coincidencia y disminuye el tiempo computacional. De esta manera, se soluciona el problema de la estimacion de la posicion basandose en la descomposicion de las homografias mundo a imagen e imagen a imagen. (II) Deteccion obstaculos y elusion colisiones, donde el UAV es capaz de percibir y detectar los obstaculos frontales que se encuentran en su camino. El algoritmo de deteccion imita comportamientos humanos para detectar los obstaculos que se acercan, mediante el analisis de la magnitud del cambio de los puntos caracteristicos detectados de referencia, combinado con los ratios de expansion de los contornos convexos construidos alrededor de los puntos caracteristicos detectados en frames consecutivos. A continuacion, comparando la proporcion del area del obstaculo y la posicion del UAV, el metodo decide si el obstaculo detectado puede provocar una colision. Por ultimo, el algoritmo extrae las zonas libres de colision alrededor del obstaculo y combinandolo con los puntos de referencia, elUAV realiza la maniobra de evasion. (III) Guiado de navegacion, que genera los puntos de referencia para determinar la trayectoria de vuelo basada en el entorno y en los obstaculos detectados que encuentra. Proporciona una estrategia para seguir los segmentos del trazado de una manera eficiente y realizar la maniobra de vuelo con suavidad. (IV) Guiado por Vision, para ofrecer soluciones de control diferentes (Control de Logica Fuzzy (FLC) y PID), basados en la informacion visual obtenida con el fin de lograr la estabilidad de vuelo, asi como realizar la maniobra correcta para evitar posibles colisiones y seguir los puntos de referencia. Todos los algoritmos propuestos han sido verificados con vuelos reales en ambientes exteriores e interiores, tomando en consideracion condiciones visuales como la iluminacion y las texturas. Los resultados obtenidos han sido validados con otros sistemas: como el sistema de captura de movimiento VICON y DGPS en el caso del algoritmo de estimacion de la posicion y orientacion. Ademas, los algoritmos propuestos han sido comparados con trabajos anteriores recogidos en el estado del arte con resultados que demuestran una mejora de la precision y la robustez de los algoritmos propuestos. Esta tesis doctoral concluye que los sensores visuales tienen las ventajes de tener un peso ligero y un bajo consumo y, proporcionar informacion fiable, lo cual lo hace una poderosa herramienta en los sistemas de navegacion para aumentar la autonomia de los UAVs en aplicaciones del mundo real.Programa Oficial de Doctorado en Ingeniería Eléctrica, Electrónica y AutomáticaPresidente: Carlo Regazzoni.- Secretario: Fernando García Fernández.- Vocal: Pascual Campoy Cerver

Remote Run-Time Failure Detection and Recovery Control For Quadcopters

We propose an adaptive run-time failure recovery control system for quadcopter drones, based on remote real-time processing of measurement data streams. Particularly, the measured RPM values of the quadcopter motors are transmitted to a remote machine which hosts failure detection algorithms and performs recovery procedure. The proposed control system consists of three distinct parts: (1) A set of computationally simple PID controllers locally onboard the drone, (2) a set of computationally more demanding remotely hosted algorithms for real-time drone state detection, and (3) a digital twin co-execution software platform — the ModelConductor-eXtended — for two-way signal data exchange between the former two. The local on-board control system is responsible for maneuvering the drone in all conditions: path tracking under normal operation and safe landing in a failure state. The remote control system, on the other hand, is responsible for detecting the state of the drone and communicating the corresponding control commands and controller parameters to the drone in real time. The proposed control system concept is demonstrated via simulations in which a drone is represented by the widely studied Quad-Sim six degrees-of-freedom Simulink model. Results show that the trained failure detection binary classifier achieves a high level of performance with F1-score of 96.03%. Additionally, time analysis shows that the proposed remote control system, with average execution time of 0.49 milliseconds and total latency of 6.92 milliseconds in two-way data communication link, meets the real-time constraints of the problem. The potential practical applications for the presented approach are in drone operation in complex environments such as factories (indoor) or forests (outdoor). </p

Position Control of an Unmanned Aerial Vehicle From a Mobile Ground Vehicle

Quadcopters have been developed with controls providing good maneuverability, simple mechanics, and the ability to hover, take-off and land vertically with precision. Due to their small size, they can get close to targets of interest and furthermore stay undetected at lower heights. The main drawbacks of a quadcopter are its high-power consumption and payload restriction, due to which, the number of onboard sensors is constrained. To overcome this limitation, vision-based localization techniques and remote control for the quadcopter are essential areas of current research. The core objective of this research is to develop a closed loop feedback system between an Unmanned Aerial Vehicle (UAV) and a mobile ground vehicle. With this closed loop system, the moving ground vehicle aims to navigate the UAV remotely. The ground vehicle uses a pure pursuit algorithm to traverse a pre-defined path. A Proportional-Integral-Derivative (PID) controller is actualized for position control and attitude stabilization of the UAV. The issue of tracking and 3D pose-estimation of the UAV in light of vision sensing is explored. An estimator to track the states of the UAV, utilizing the images obtained from a single camera mounted on the ground vehicle is developed. This estimator coupled with a Kalman filter determines the UAV\u27s three dimensional position. The relative position of the UAV with the moving ground vehicle and the control output from a joint centralized PD controller is used to navigate the UAV and follow the motion of the ground vehicle in closed loop to avoid time delays. This closed loop system is simulated in MATLAB and Simulink to validate the proposed control and estimation approach. The results obtained validate the control architecture proposed to attain closed loop feedback between the UAV and the mobile ground vehicle