NU-AIR -- A Neuromorphic Urban Aerial Dataset for Detection and Localization of Pedestrians and Vehicles

This paper presents an open-source aerial neuromorphic dataset that captures pedestrians and vehicles moving in an urban environment. The dataset, titled NU-AIR, features 70.75 minutes of event footage acquired with a 640 x 480 resolution neuromorphic sensor mounted on a quadrotor operating in an urban environment. Crowds of pedestrians, different types of vehicles, and street scenes featuring busy urban environments are captured at different elevations and illumination conditions. Manual bounding box annotations of vehicles and pedestrians contained in the recordings are provided at a frequency of 30 Hz, yielding 93,204 labels in total. Evaluation of the dataset's fidelity is performed through comprehensive ablation study for three Spiking Neural Networks (SNNs) and training ten Deep Neural Networks (DNNs) to validate the quality and reliability of both the dataset and corresponding annotations. All data and Python code to voxelize the data and subsequently train SNNs/DNNs has been open-sourced.Comment: 20 pages, 5 figure

Mathematical programming for multi-vehicle motion planning under communication constraints

Multi-Vehicle Motion Planning (MVMP) problems feature multiple vehicles traversing in their work space while avoiding collisions with each other and with other obstacles. Real world MVMP problems require the optimization of suitable performance measures under an array of constraints including kinematics, dynamics, communication connectivity, target tracking, and collision avoidance. The general MVMP problem can thus be formulated as a mathematical program (MP). In this thesis we present a mathematical programming framework that captures the salient features of the general MVMP problem. We use state-of-the-art solution algorithms and associated numerical solvers developed by our group to solve MVMP problems using this framework.To demonstrate the effectiveness of this framework, we investigate a variant of the general MVMP problem, viz. Multi-Vehicle Path Coordination wherein we generate time optimal velocity profiles for multiple robotic vehicles confined to move along predetermined and fixed paths. Each robot must follow a fixed and known path, arrive at its goal as quickly as possible (or at least not increase the time for the last robot to arrive at its goal) and stay in communication with other robots in the arena throughout its journey. We enforce a variety of communication connectivity constraints that incorporate deterministic and stochastic physical layer communication models to ensure that the robots can communicate with each other while in transit. We develop Partition Elimination constraints that assist in ensuring that the communication network is fully connected while the robots are in transit. The resulting mathematical programming models are solved using state-of-the-art highly efficient mixed integer nonlinear optimization tools developed by our group. Several conditions that affect the feasibility of the problem are identified and formalized.Both centralized and decentralized formulations are studied to demonstrate (i) the effect of communication connectivity requirements on robot velocity profiles; (ii) the dependence of the scenario completion time on communication connectivity requirements; (iii) the dependence of computation time on the number of robots; (iv) the tradeoff between the arrival time and the communication connectivity requirements. As MP solution algorithms and associated numerical solvers continue to develop, we anticipate that MP solution techniques will be applied to an increasing number of MVMP problems and this thesis may serve as a guide for future MVMP research.Ph.D., Electrical Engineering -- Drexel University, 201

Event-Based Motion Capture System for Online Multi-Quadrotor Localization and Tracking

Motion capture systems are crucial in developing multi-quadrotor systems due to their ability to provide fast and accurate ground truth measurements for tracking and control. This paper presents the implementation details and experimental validation of a relatively low-cost motion-capture system for multi-quadrotor motion planning using an event camera. The real-time, multi-quadrotor detection and tracking tasks are performed using a deep learning network You-Only-Look-Once (YOLOv5) and a k-dimensional (k-d) tree, respectively. An optimization-based decentralized motion planning algorithm is implemented to demonstrate the effectiveness of this motion capture system. Extensive experimental evaluations were performed to (1) compare the performance of four deep-learning algorithms for high-speed multi-quadrotor detection on event-based data, (2) study precision, recall, and F1 scores as functions of lighting conditions and camera motion, and (3) investigate the scalability of this system as a function of the number of quadrotors flying in the arena. Comparative analysis of the deep learning algorithms on a consumer-grade GPU demonstrates a 4.8× to 12× sampling/inference rate advantage that YOLOv5 provides over representative one- and two-stage detectors and a 1.14× advantage over YOLOv4. In terms of precision and recall, YOLOv5 performed 15% to 18% and 27% to 41% better than representative state-of-the-art deep learning networks. Graceful detection and tracking performance degradation was observed in the face of progressively darker ambient light conditions. Despite severe camera motion, YOLOv5 precision and recall values of 94% and 98% were achieved, respectively. Finally, experiments involving up to six indoor quadrotors demonstrated the scalability of this approach. This paper also presents the first open-source event camera dataset in the literature, featuring over 10,000 fully annotated images of multiple quadrotors operating in indoor and outdoor environments

Distributed Motion Planning for Multiple Quadrotors in Presence of Wind Gusts

This work demonstrates distributed motion planning for multi-rotor unmanned aerial vehicle in a windy outdoor environment. The motion planning is modeled as a receding horizon mixed integer nonlinear programming (RH-MINLP) problem. Each quadrotor solves an RH-MINLP to generate its time-optimal speed profile along a minimum snap spline path while satisfying constraints on kinematics, dynamics, communication connectivity, and collision avoidance. The presence of wind disturbances causes the motion planner to continuously regenerate new motion plans, thereby significantly increasing the computational time and possibly leading to safety violations. Control Barrier Functions (CBFs) are used for assist in collision avoidance in the face of wind disturbances while alleviating the need to recalculate the motion plans continually. The RH-MINLPs are solved using a novel combination of heuristic and optimal methods, namely Simulated Annealing and interior-point methods, respectively, to handle discrete variables and nonlinearities in real-time feasibly. The framework is validated in simulations featuring up to 50 quadrotors and Hardware-in-the-loop (HWIL) experiments, followed by outdoor field tests featuring up to 6 DJI M100 quadrotors. Results demonstrate (1) fast online motion planning for outdoor communication-centric multi-quadrotor operations and (2) the utility of CBFs in providing effective motion plans