Near-Optimal Control of a Quadcopter Using Reinforcement Learning

This paper presents a novel control method for quadcopters that achieves near-optimal tracking control for input-affine nonlinear quadcopter dynamics. The method uses a reinforcement learning algorithm called Single Network Adaptive Critics (SNAC), which approximates a solution to the discrete-time Hamilton-Jacobi-Bellman (DT-HJB) equation using a single neural network trained offline. The control method involves two SNAC controllers, with the outer loop controlling the linear position and velocities (position control) and the inner loop controlling the angular position and velocities (attitude control). The resulting quadcopter controller provides optimal feedback control and tracks a trajectory for an infinite-horizon, and it is compared with commercial optimal control software. Furthermore, the closed-loop controller can control the system with any initial conditions within the domain of training. Overall, this research demonstrates the benefits of using SNAC for nonlinear control, showing its ability to achieve near-optimal tracking control while reducing computational complexity. This paper provides insights into a new approach for controlling quadcopters, with potential applications in various fields such as aerial surveillance, delivery, and search and rescue

COORDINATION OF LEADER-FOLLOWER MULTI-AGENT SYSTEM WITH TIME-VARYING OBJECTIVE FUNCTION

This thesis aims to introduce a new framework for the distributed control of multi-agent systems with adjustable swarm control objectives. Our goal is twofold: 1) to provide an overview to how time-varying objectives in the control of autonomous systems may be applied to the distributed control of multi-agent systems with variable autonomy level, and 2) to introduce a framework to incorporate the proposed concept to fundamental swarm behaviors such as aggregation and leader tracking. Leader-follower multi-agent systems are considered in this study, and a general form of time-dependent artificial potential function is proposed to describe the varying objectives of the system in the case of complete information exchange. Using Lyapunov methods, the stability and boundedness of the agents\u27 trajectories under single order and higher order dynamics are analyzed. Illustrative numerical simulations are presented to demonstrate the validity of our results. Then, we extend these results for multi-agent systems with limited information exchange and switching communication topology. The first steps of the realization of an experimental framework have been made with the ultimate goal of verifying the simulation results in practice

Online Deep Learning for Improved Trajectory Tracking of Unmanned Aerial Vehicles Using Expert Knowledge

This work presents an online learning-based control method for improved trajectory tracking of unmanned aerial vehicles using both deep learning and expert knowledge. The proposed method does not require the exact model of the system to be controlled, and it is robust against variations in system dynamics as well as operational uncertainties. The learning is divided into two phases: offline (pre-)training and online (post-)training. In the former, a conventional controller performs a set of trajectories and, based on the input-output dataset, the deep neural network (DNN)-based controller is trained. In the latter, the trained DNN, which mimics the conventional controller, controls the system. Unlike the existing papers in the literature, the network is still being trained for different sets of trajectories which are not used in the training phase of DNN. Thanks to the rule-base, which contains the expert knowledge, the proposed framework learns the system dynamics and operational uncertainties in real-time. The experimental results show that the proposed online learning-based approach gives better trajectory tracking performance when compared to the only offline trained network.Comment: corrected version accepted for ICRA 201

A Model-Free Control Algorithm Based on the Sliding Mode Control Method with Applications to Unmanned Aircraft Systems

Control methods require the use of a system model for the design and tuning of the controllers in meeting and/or exceeding the control system performance objectives. However, system models contain errors and uncertainties that also may be complex to develop and to generalize for a large class of systems such as those for unmanned aircraft systems. In particular, the sliding control method is a superior robust nonlinear control approach due to the direct handling of nonlinearities and uncertainties that can be used in tracking problems for unmanned aircraft system. However, the derivation of the sliding mode control law is tedious since a unique and distinct control law needs to be derived for every individual system and cannot be applied to general systems that may encompass all classifications of unmanned aircraft systems. In this work, a model-free control algorithm based on the sliding mode control method is developed and generalized for all classes of unmanned aircraft systems used in robust tracking control applications. The model-free control algorithm is derived with knowledge of the system’s order, state measurements, and control input gain matrix shape and bounds and is not dependent on a mathematical system model. The derived control law is tested using a high-fidelity simulation of a quadrotor-type unmanned aircraft system and the results are compared to a traditional linear controller for tracking performance and power consumption. Realistic type hardware inputs from joysticks and inertial measurement units were simulated for the analysis. Finally, the model-free control algorithm was implemented on a quadrotor-type unmanned aircraft system testbed used in real flight experimental testing. The experimental tracking performance and power consumption was analyzed and compared to a traditional linear-type controller. Results showed that the model-free approach is superior in tracking performance and power consumption compared to traditional linear-type control strategies

Reinforcement Learning for UAV Attitude Control

Autopilot systems are typically composed of an "inner loop" providing stability and control, while an "outer loop" is responsible for mission-level objectives, e.g. way-point navigation. Autopilot systems for UAVs are predominately implemented using Proportional, Integral Derivative (PID) control systems, which have demonstrated exceptional performance in stable environments. However more sophisticated control is required to operate in unpredictable, and harsh environments. Intelligent flight control systems is an active area of research addressing limitations of PID control most recently through the use of reinforcement learning (RL) which has had success in other applications such as robotics. However previous work has focused primarily on using RL at the mission-level controller. In this work, we investigate the performance and accuracy of the inner control loop providing attitude control when using intelligent flight control systems trained with the state-of-the-art RL algorithms, Deep Deterministic Gradient Policy (DDGP), Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO). To investigate these unknowns we first developed an open-source high-fidelity simulation environment to train a flight controller attitude control of a quadrotor through RL. We then use our environment to compare their performance to that of a PID controller to identify if using RL is appropriate in high-precision, time-critical flight control.Comment: 13 pages, 9 figure

Development of c-means Clustering Based Adaptive Fuzzy Controller for A Flapping Wing Micro Air Vehicle

Advanced and accurate modelling of a Flapping Wing Micro Air Vehicle (FW MAV) and its control is one of the recent research topics related to the field of autonomous Unmanned Aerial Vehicles (UAVs). In this work, a four wing Natureinspired (NI) FW MAV is modeled and controlled inspiring by its advanced features like quick flight, vertical take-off and landing, hovering, and fast turn, and enhanced manoeuvrability when contrasted with comparable-sized fixed and rotary wing UAVs. The Fuzzy C-Means (FCM) clustering algorithm is utilized to demonstrate the NIFW MAV model, which has points of interest over first principle based modelling since it does not depend on the system dynamics, rather based on data and can incorporate various uncertainties like sensor error. The same clustering strategy is used to develop an adaptive fuzzy controller. The controller is then utilized to control the altitude of the NIFW MAV, that can adapt with environmental disturbances by tuning the antecedent and consequent parameters of the fuzzy system.Comment: this paper is currently under review in Journal of Artificial Intelligence and Soft Computing Researc