Using Actor-Critic Reinforcement Learning for Control of a Quadrotor Dynamics

This paper presents a quadrotor controller using reinforcement learning to generate near-optimal control signals. Two actor-critic algorithms are trained to control quadrotor dynamics. The dynamics are further simplified using small angle approximation. The actor-critic algorithm’s control policy is derived from Bellman’s equation providing a sufficient condition to optimality. Additionally, a smoother converter is implemented into the trajectory providing more reliable results. This paper provides derivations to the quadrotor’s dynamics and explains the control using the actor-critic algorithm. The results and simulations are compared to solutions from a commercial, optimal control solver, called DIDO

Controlled manipulation using autonomous aerial systems

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Mechanical Engineering, February 2013.Cataloged from PDF version of thesis.Includes bibliographical references (p. 134-135).The main focus of the thesis is to design and control Autonomous Aerial Systems, also referred to as Unmanned Aerial Vehicles (UAVs). UAVs are able to hover and navigate in space using the thrust forces generated by the propellers. One of the simplest such vehicles that is widely used is a Quadrotor. While UAVs have been predominantly used for "fly and sense" applications, very few investigations have focused on using them to perform manipulation by contact. The latter is challenging because of the dual goal of performing manipulation and maintaining stable flight. Because Quadrotors can quickly reach a location, their ability to manipulate can be impactful in many scenarios. While efficient flight control of Quadrotor has been an active research area, using Quadrotor to perform manipulation is novel and challenging. In this thesis, a range of Quadrotor designs and control strategies are proposed in order to carry out autonomous manipulation of objects. We first derive a dynamic model of the Quadrotor that accounts for the presence of contact, object dynamics and kinematics. To improve manipulation performance, a passive light-weight end-effector interface between the Quadrotor and the object is proposed. The complexity of the dynamics is systematically reduced by making certain assumptions. The resulting dynamic model is divided into nonlinear subsystems on the basis of their degrees of freedom, for each of which separate controllers are designed. An efficient docking approach is proposed that permits fast and aggressive docking, even at very high speeds. Because a single Quadrotor UAS is limited in manipulation capability, a multi Quadrotor cooperative manipulation scheme is proposed. Control strategies are proposed to deal with kinematic and parametric uncertainties. A manipulation scheme to open a door with unknown hinge location is proposed. A nonlinear adaptive controller is implemented to perform efficient tracking in the presence of parametric uncertainty. In order to improve robustness to accidental contacts, a novel flexible Quadrotor, denoted as ParaFlex, is designed. The advantages of ParaFlex over a rigid Quadrotor are demonstrated. A Simulation, Test and Validation Environment (STeVE) is developed to facilitate smooth and efficient transition from design process to simulation to experiments.by Manohar B. Srikanth.Ph.D

Safe and accurate MAV Control, navigation and manipulation

This work focuses on the problem of precise, aggressive and safe Micro Aerial Vehicle (MAV) navigation as well as deployment in applications which require physical interaction with the environment. To address these issues, we propose three different MAV model based control algorithms that rely on the concept of receding horizon control. As a starting point, we present a computationally cheap algorithm which utilizes an approximate linear model of the system around hover and is thus maximally accurate for slow reference maneuvers. Aiming at overcoming the limitations of the linear model parameterisation, we present an extension to the first controller which relies on the true nonlinear dynamics of the system. This approach, even though computationally more intense, ensures that the control model is always valid and allows tracking of full state aggressive trajectories. The last controller addresses the topic of aerial manipulation in which the versatility of aerial vehicles is combined with the manipulation capabilities of robotic arms. The proposed method relies on the formulation of a hybrid nonlinear MAV-arm model which also takes into account the effects of contact with the environment. Finally, in order to enable safe operation despite the potential loss of an actuator, we propose a supervisory algorithm which estimates the health status of each motor. We further showcase how this can be used in conjunction with the nonlinear controllers described above for fault tolerant MAV flight. While all the developed algorithms are formulated and tested using our specific MAV platforms (consisting of underactuated hexacopters for the free flight experiments, hexacopter-delta arm system for the manipulation experiments), we further discuss how these can be applied to other underactuated/overactuated MAVs and robotic arm platforms. The same applies to the fault tolerant control where we discuss different stabilisation techniques depending on the capabilities of the available hardware. Even though the primary focus of this work is on feedback control, we thoroughly describe the custom hardware platforms used for the experimental evaluation, the state estimation algorithms which provide the basis for control as well as the parameter identification required for the formulation of the various control models. We showcase all the developed algorithms in experimental scenarios designed to highlight the corresponding strengths and weaknesses as well as show that the proposed methods can run in realtime on commercially available hardware.Open Acces

Modeling, identification and navigation of autonomous air vehicles

The main interest of this work is autonomous navigation of autonomous air vehicles, specifically quadrotor helicopters (quadrocopters), and the focus is on convergence to a target destination with collision avoidance. The controller computes a collision-free path leading to the target position and is based on a navigation function approach and waypoints are followed exploiting PID controller

Optimal guidance and control of heterogeneous swarms for in-orbit self-assembly of large space structures: Algorithms and experiments

Satellite design has been harshly constrained by surviving entry into space though the majority of the satellite's lifetime exists in much calmer conditions. Significant study has recently gone into assembling satellites and space structures in-orbit. Several methods have been proposed involving an assembler robot or astronaut which puts the parts together, but in the interest of saving resources we believe that it is advantageous to make this process autonomous and robust by leveraging existing optimal guidance and control schemes for a self-assembling swarm. This approach avoids single-point failures, requires significantly less ground support, provides increased reliability due to redundancy, increased flexibility, the ability to reconfigure for future missions, and the ability to self-repair. Since the satellites required could be mass-produced from a small set of different component types, the benefit from economy of scale would reduce the overall mission cost when compared to monolithic satellites. This dissertation details an optimal guidance and control scheme to enable in-orbit self-assembly of a large structure from a heterogeneous swarm of satellites. In the proposed scheme, the component satellites for the heterogeneous swarm are chosen to promote flexibility in final shape inspired by crystal structures and Islamic tile art. After the ideal fundamental building blocks are selected, basic nanosatellite-class satellite designs are presented to enable accurate attitude control simulations. The Swarm Orbital Construction Algorithm (SOCA) is a guidance and control algorithm that allows for the limited type heterogeneity and docking ability required for in-orbit assembly. The algorithm was tested in a simulated perturbed 6-DOF spacecraft dynamic environment for planar and out-of-plane final structures. The algorithm is then experimentally validated coarsely on omnidirectional wheeled robots and precisely on-board the M-STAR robots in the precision flat floor facility in the Caltech Aerospace Robotics and Control lab, the largest of its kind at any university. In support of this effort, a better way of handling nonlinear dynamics constraints within sequential convex programs was developed. SCP is a useful tool in obtaining real-time solutions to direct optimal control, but it is unable to adequately model nonlinear dynamics due to the linearization and discretization required. As nonlinear program solvers are not yet functioning in real-time, a tool is needed to bridge the gap between satisfying the nonlinear dynamics and completing execution fast enough to be useful. Two methods are proposed, sequential convex programming with nonlinear dynamics correction (SCPn) and modified SCPn (M-SCPn), which mixes SCP and SCPn to reduce runtime and improve algorithmic robustness. Both methods are proven to generate optimal state and control trajectories that satisfy the nonlinear dynamics. Simulations are presented to validate the efficacy of the methods as compared to SCP. In addition, several autonomous rendezvous and docking (AR&D) technologies were studied because in-orbit self-assembly requires repeated, reliable autonomous docking to ensure success. Docking small satellites in space is a high-risk operation due to the uncertainty in relative position and orientation and the lack of mature docking technologies. This is particularly true for missions that involve multiple docking and undocking procedures like swarm-based construction and reconfiguration. A tether-based docking system was evaluated in simulation as compared to traditional propulsive methods. The tether-based method provides a way to reduce the risk of the dock, since the docking maneuver is performed with a much smaller satellite and the reeling maneuver can be done gently. Tether-based methods still require some actuation on the docking end of the tether, and propulsion on such small systems is inexact. An electromagnetic docking system was investigated to address these issues. Designed with reconfigurable self-assembly in mind, the gripping mechanism is androgynous, able to dock at a variety of relative orientations, and tolerant of small misalignments. The electromagnetic system can be used either on the end of a tether or on the main spacecraft itself since the electromagnet is well controlled and the measurement of the ambient electromagnetic field can be used as to improve the intersatellite distance estimate enough to reduce the risk of docking to the main spacecraft. The performance of this system was validated experimentally on-board the M-STARs. The performance of the electromagnetic docking system on-board the simulators is then compared against a propulsive docking system tested in the same way. Overall, this dissertation provides optimal guidance and control algorithms for nonlinear systems to enable in-orbit self-assembly of heterogeneous swarms

Proceedings of the International Micro Air Vehicles Conference and Flight Competition 2017 (IMAV 2017)

The IMAV 2017 conference has been held at ISAE-SUPAERO, Toulouse, France from Sept. 18 to Sept. 21, 2017. More than 250 participants coming from 30 different countries worldwide have presented their latest research activities in the field of drones. 38 papers have been presented during the conference including various topics such as Aerodynamics, Aeroacoustics, Propulsion, Autopilots, Sensors, Communication systems, Mission planning techniques, Artificial Intelligence, Human-machine cooperation as applied to drones

Prohibited Volume Avoidance for Aircraft

This thesis describes the development of a pilot override control system that prevents aircraft entering critical regions of space, known as prohibited volumes. The aim is to prevent another 9/11 style terrorist attack, as well as act as a general safety system for transport aircraft. The thesis presents the design and implementation of three core modules in the system; the trajectory generation algorithm, the trigger mechanism for the pilot override and the trajectory following element. The trajectory generation algorithm uses a direct multiple shooting strategy to provide trajectories through online computation that avoid pre-defi ned prohibited volume exclusion regions, whilst accounting for the manoeuvring capabilities of the aircraft. The trigger mechanism incorporates the logic that decides the time at which it is suitable for the override to be activated, an important consideration for ensuring that the system is not overly restrictive for a pilot. A number of methods are introduced, and for safety purposes a composite trigger that incorporates di fferent strategies is recommended. Trajectory following is best achieved via a nonlinear guidance law. The guidance logic sends commands in pitch, roll and yaw to the control surfaces of the aircraft, in order to closely follow the generated avoidance trajectory. Testing and validation is performed using a full motion simulator, with volunteers flying a representative aircraft model and attempting to penetrate prohibited volumes. The proof-of-concept system is shown to work well, provided that extreme aircraft manoeuvres are prevented near the exclusion regions. These hard manoeuvring envelope constraints allow the trajectory following controllers to follow avoidance trajectories accurately from an initial state within the bounding set. In order to move the project closer to a commercial product, operator and regulator input is necessary, particularly due to the radical nature of the pilot override system

Aerial Vehicles

This book contains 35 chapters written by experts in developing techniques for making aerial vehicles more intelligent, more reliable, more flexible in use, and safer in operation.It will also serve as an inspiration for further improvement of the design and application of aeral vehicles. The advanced techniques and research described here may also be applicable to other high-tech areas such as robotics, avionics, vetronics, and space