Collision avoidance interface for safe piloting of unmanned vehicles using a mobile device

Autonomous robots and vehicles can perform tasks that are unsafe or undesirable for humans to do themselves, such as investigate safety in nuclear reactors or assess structural damage to a building or bridge after an earthquake. In addition, improvements in autonomous modes of such vehicles are making it easier for minimally-trained individuals to operate the vehicles. As the autonomous capabilities advance, the user's role shifts from a direct teleoperator to a supervisory control role. Since the human operator is often better suited to make decisions in uncertain situations, it is important for the human operator to have awareness of the environment in which the vehicle is operating in order to prevent collisions and damage to the vehicle as well as the structures and people in the vicinity. In this paper, we present the Collision and Obstacle Detection and Alerting (CODA) display, a novel interface to enable safe piloting of a Micro Aerial Vehicle with a mobile device in real-world settings.Boeing Compan

Exploring Alternative Control Modalities for Unmanned Aerial Vehicles

Unmanned aerial vehicles (UAVs), commonly known as drones, are defined by the International Civil Aviation Organization (ICAO) as an aircraft without a human pilot on board. They are currently utilized primarily in the defense and security sectors but are moving towards the general market in surprisingly powerful and inexpensive forms. While drones are presently restricted to non-commercial recreational use in the USA, it is expected that they will soon be widely adopted for both commercial and consumer use. Potentially, UAVs can revolutionize various business sectors including private security, agricultural practices, product transport and maybe even aerial advertising. Business Insider foresees that 12% of the expected $98 billion cumulative global spending on aerial drones through the following decade will be for business purposes.[28] At the moment, most drones are controlled by some sort of classic joystick or multitouch remote controller. While drone manufactures have improved the overall controllability of their products, most drones shipped today are still quite challenging for inexperienced users to pilot. In order to help mitigate the controllability challenges and flatten the learning curve, gesture controls can be utilized to improve piloting UAVs. The purpose of this study was to develop and evaluate an improved and more intuitive method of flying UAVs by supporting the use of hand gestures, and other non-traditional control modalities. The goal was to employ and test an end-to-end UAV system that provides an easy-to-use control interface for novice drone users. The expectation was that by implementing gesture-based navigation, the novice user will have an overall enjoyable and safe experience quickly learning how to navigate a drone with ease, and avoid losing or damaging the vehicle while they are on the initial learning curve. During the course of this study we have learned that while this approach does offer lots of promise, there are a number of technical challenges that make this problem much more challenging than anticipated. This thesis details our approach to the problem, analyzes the user data we collected, and summarizes the lessons learned

Software platform to control squads of unmanned vehicles in realtime

Unmanned Aerial Vehicles (UAVs) applications are becoming more and more researched. “Drones” (UAVs) were mainly used as a military technology but are now becoming a leisure and professional activity for many civilian users. Nowadays UAVs are mostly controlled by the use of a controller that operates in Radio Control (RC), although this method of communication limits the vehicle’s distance to the line of sight of the operator. As a need to overcome the line of sight obstacle, cellular networks provide a mean of connection and as the coverage is increasing they’re a natural solution as Wi-Fi is not present everywhere. In order to accomplish this communication between Drone and Operator, there needs to be a Ground Control Station that provides the user all the tools needed to operate the vehicle. This project provides a software platform that is able to monitor a squad of drones whilst also being able to control one at a time. The platform maintains the communication with the vehicle at all times, and is also be able to receive live-video in order to overcome the beyond line of sight obstacle. Besides this, the application provides an admin user, with the capability of overriding a regular user’s control, assigning the user’s drone to itself for controlling purposes. A public server is used to make the exchanging of messages possible, and to have a centralized control over drones and their respective user. Keywords:Os Veículos Aéreos Não Tripulados (UAVs) são cada vez mais utilizados e desenvolvidos. O que antes era utilizado principalmente como tecnologia militar, tem-se vindo a tornar uma profissão ou um hobbie para muitos civis. Hoje em dia os UAVs são controlados geralmente através de um comando, que opera em Radio Controlo (RC) e, embora seja muito utilizado, este método de comunicação limita a distância do veículo à linha de visão do operador. Este é um obstáculo que se tem procurado ultrapassar e as redes móveis providenciam o meio necessário para tal. Desta forma e como a cobertura das redes móveis tem aumentado progressivamente é hoje em dia uma alternativa ao Wi-Fi que não tem o mesmo alcance nem a mesma cobertura. Para que a comunicação entre drone e operador seja viável, tem que existir uma estação de controlo que forneça ao utilizador todas as ferramentas necessárias para operar o veículo. Este projeto visa a criação de uma plataforma de software que seja capaz de monitorizar uma esquadra de UAVs e seja também capaz de controlar um aparelho de cada vez. A plataforma mantém a comunicação com o veículo em todos os momentos, e permite ainda a receção de vídeo ao vivo, superando assim o obstáculo da linha de vista. Também é disponibilizada a um administrador a capacidade de retirar o controlo dos utilizadores aos seus drones alterando assim o responsável pelo controlo. É também utilizado um servidor público de forma a tornar a troca de mensagens possível e também por outro lado, controlar de forma centralizada os drones e os seus respetivos utilizadores. Palavras-chave: Monitorização, Controlo Remoto, Redes Sem fios, Aplicação, Drone

Using learning from demonstration to enable automated flight control comparable with experienced human pilots

Modern autopilots fall under the domain of Control Theory which utilizes Proportional Integral Derivative (PID) controllers that can provide relatively simple autonomous control of an aircraft such as maintaining a certain trajectory. However, PID controllers cannot cope with uncertainties due to their non-adaptive nature. In addition, modern autopilots of airliners contributed to several air catastrophes due to their robustness issues. Therefore, the aviation industry is seeking solutions that would enhance safety. A potential solution to achieve this is to develop intelligent autopilots that can learn how to pilot aircraft in a manner comparable with experienced human pilots. This work proposes the Intelligent Autopilot System (IAS) which provides a comprehensive level of autonomy and intelligent control to the aviation industry. The IAS learns piloting skills by observing experienced teachers while they provide demonstrations in simulation. A robust Learning from Demonstration approach is proposed which uses human pilots to demonstrate the task to be learned in a flight simulator while training datasets are captured. The datasets are then used by Artificial Neural Networks (ANNs) to generate control models automatically. The control models imitate the skills of the experienced pilots when performing the different piloting tasks while handling flight uncertainties such as severe weather conditions and emergency situations. Experiments show that the IAS performs learned skills and tasks with high accuracy even after being presented with limited examples which are suitable for the proposed approach that relies on many single-hidden-layer ANNs instead of one or few large deep ANNs which produce a black-box that cannot be explained to the aviation regulators. The results demonstrate that the IAS is capable of imitating low-level sub-cognitive skills such as rapid and continuous stabilization attempts in stormy weather conditions, and high-level strategic skills such as the sequence of sub-tasks necessary to takeoff, land, and handle emergencies

NtoM: a concept of operations for pilots of multiple remotely piloted aircraft

The concept of operations proposed here pursues the feasibility, from a human factors perspective, of having a single pilot/aircrew controlling several remotely piloted aircraft systems at once in non-segregated airspace. To meet such feasibility, this multitasking must be safe and not interfere with the job of the air traffic controllers due to delays or errors associated with parallel piloting. To that end, a set of measures at several levels is suggested, which includes workload prediction and balance, pilot activity monitoring, and a special emphasis on interface usability and the pilot’s situational awareness. The concept relies greatly on the exploitation of the potential of Controller-Pilot Data Link Communications, anticipating future widespread implementation and full use. Experiments comparing the performance of the same pseudo-pilots before and after the implementation of part of the measures showed a decrease in the number of errors, oversights and subjective stress.Peer ReviewedPostprint (published version

Outdoor operations of multiple quadrotors in windy environment

Coordinated multiple small unmanned aerial vehicles (sUAVs) offer several advantages over a single sUAV platform. These advantages include improved task efficiency, reduced task completion time, improved fault tolerance, and higher task flexibility. However, their deployment in an outdoor environment is challenging due to the presence of wind gusts. The coordinated motion of a multi-sUAV system in the presence of wind disturbances is a challenging problem when considering collision avoidance (safety), scalability, and communication connectivity. Performing wind-agnostic motion planning for sUAVs may produce a sizeable cross-track error if the wind on the planned route leads to actuator saturation. In a multi-sUAV system, each sUAV has to locally counter the wind disturbance while maintaining the safety of the system. Such continuous manipulation of the control effort for multiple sUAVs under uncertain environmental conditions is computationally taxing and can lead to reduced efficiency and safety concerns. Additionally, modern day sUAV systems are susceptible to cyberattacks due to their use of commercial wireless communication infrastructure. This dissertation aims to address these multi-faceted challenges related to the operation of outdoor rotor-based multi-sUAV systems. A comprehensive review of four representative techniques to measure and estimate wind speed and direction using rotor-based sUAVs is discussed. After developing a clear understanding of the role wind gusts play in quadrotor motion, two decentralized motion planners for a multi-quadrotor system are implemented and experimentally evaluated in the presence of wind disturbances. The first planner is rooted in the reinforcement learning (RL) technique of state-action-reward-state-action (SARSA) to provide generalized path plans in the presence of wind disturbances. While this planner provides feasible trajectories for the quadrotors, it does not provide guarantees of collision avoidance. The second planner implements a receding horizon (RH) mixed-integer nonlinear programming (MINLP) model that is integrated with control barrier functions (CBFs) to guarantee collision-free transit of the multiple quadrotors in the presence of wind disturbances. Finally, a novel communication protocol using Ethereum blockchain-based smart contracts is presented to address the challenge of secure wireless communication. The U.S. sUAV market is expected to be worth $92 Billion by 2030. The Association for Unmanned Vehicle Systems International (AUVSI) noted in its seminal economic report that UAVs would be responsible for creating 100,000 jobs by 2025 in the U.S. The rapid proliferation of drone technology in various applications has led to an increasing need for professionals skilled in sUAV piloting, designing, fabricating, repairing, and programming. Engineering educators have recognized this demand for certified sUAV professionals. This dissertation aims to address this growing sUAV-market need by evaluating two active learning-based instructional approaches designed for undergraduate sUAV education. The two approaches leverages the interactive-constructive-active-passive (ICAP) framework of engagement and explores the use of Competition based Learning (CBL) and Project based Learning (PBL). The CBL approach is implemented through a drone building and piloting competition that featured 97 students from undergraduate and graduate programs at NJIT. The competition focused on 1) drone assembly, testing, and validation using commercial off-the-shelf (COTS) parts, 2) simulation of drone flight missions, and 3) manual and semi-autonomous drone piloting were implemented. The effective student learning experience from this competition served as the basis of a new undergraduate course on drone science fundamentals at NJIT. This undergraduate course focused on the three foundational pillars of drone careers: 1) drone programming using Python, 2) designing and fabricating drones using Computer-Aided Design (CAD) and rapid prototyping, and 3) the US Federal Aviation Administration (FAA) Part 107 Commercial small Unmanned Aerial Vehicles (sUAVs) pilot test. Multiple assessment methods are applied to examine the students’ gains in sUAV skills and knowledge and student attitudes towards an active learning-based approach for sUAV education. The use of active learning techniques to address these challenges lead to meaningful student engagement and positive gains in the learning outcomes as indicated by quantitative and qualitative assessments