Trajectory Optimization for the Handling of Elastically Coupled Objects via Reinforcement Learning and Flatness-Based Control

Positioning objects in industrial handling applications is often compromised by elasticity-induced oscillations reducing the possible motion time and thereby the performance and profitability of the automation solution. Existing approaches for oscillation reduction mostly focus on the elasticity of the handling system itself, i.e. the robot structure. Depending on the task, elastic parts or elastic grippers like suction cups strongly influence the oscillation and prevent faster positioning. In this paper, the problem is investigated exemplarily with a typical handling robot and an additional end effector setup representing the elastic load. The handling object is modeled as a base-excited spring and mass, making the proposed approach independent from the robot structure. A model-based feed-forward control based on differential flatness and a machine-learning method are used to reduce oscillations solely with a modification of the end effector trajectory of the robot. Both methods achieve a reduction of oscillation amplitudes of 85% for the test setup, promising a significant increase in performance. Further investigations on the uncertainty of the parameterization prove the applicability of the not yet widely-used learning approach in the field of oscillation reduction

High fidelity progressive reinforcement learning for agile maneuvering UAVs

In this work, we present a high fidelity model based progressive reinforcement learning method for control system design for an agile maneuvering UAV. Our work relies on a simulation-based training and testing environment for doing software-in-the-loop (SIL), hardware-in-the-loop (HIL) and integrated flight testing within photo-realistic virtual reality (VR) environment. Through progressive learning with the high fidelity agent and environment models, the guidance and control policies build agile maneuvering based on fundamental control laws. First, we provide insight on development of high fidelity mathematical models using frequency domain system identification. These models are later used to design reinforcement learning based adaptive flight control laws allowing the vehicle to be controlled over a wide range of operating conditions covering model changes on operating conditions such as payload, voltage and damage to actuators and electronic speed controllers (ESCs). We later design outer flight guidance and control laws. Our current work and progress is summarized in this work

Reinforcement Learning and Planning for Preference Balancing Tasks

Robots are often highly non-linear dynamical systems with many degrees of freedom, making solving motion problems computationally challenging. One solution has been reinforcement learning (RL), which learns through experimentation to automatically perform the near-optimal motions that complete a task. However, high-dimensional problems and task formulation often prove challenging for RL. We address these problems with PrEference Appraisal Reinforcement Learning (PEARL), which solves Preference Balancing Tasks (PBTs). PBTs define a problem as a set of preferences that the system must balance to achieve a goal. The method is appropriate for acceleration-controlled systems with continuous state-space and either discrete or continuous action spaces with unknown system dynamics. We show that PEARL learns a sub-optimal policy on a subset of states and actions, and transfers the policy to the expanded domain to produce a more refined plan on a class of robotic problems. We establish convergence to task goal conditions, and even when preconditions are not verifiable, show that this is a valuable method to use before other more expensive approaches. Evaluation is done on several robotic problems, such as Aerial Cargo Delivery, Multi-Agent Pursuit, Rendezvous, and Inverted Flying Pendulum both in simulation and experimentally. Additionally, PEARL is leveraged outside of robotics as an array sorting agent. The results demonstrate high accuracy and fast learning times on a large set of practical applications

A Hybrid Control Approach for the Swing Free Transportation of a Double Pendulum with a Quadrotor

In this article, a control strategy approach is proposed for a system consisting of a quadrotor transporting a double pendulum. In our case, we attempt to achieve a swing free transportation of the pendulum, while the quadrotor closely follows a specific trajectory. This dynamic system is highly nonlinear, therefore, the fulfillment of this complex task represents a demanding challenge. Moreover, achieving dampening of the double pendulum oscillations while following a precise trajectory are conflicting goals. We apply a proportional derivative (PD) and a model predictive control (MPC) controllers for this task. Transportation of a multiple pendulum with an aerial robot is a step forward in the state of art towards the study of the transportation of loads with complex dynamics. We provide the modeling of the quadrotor and the double pendulum. For MPC we define the cost function that has to be minimized to achieve optimal control. We report encouraging positive results on a simulated environmentcomparing the performance of our MPC-PD control circuit against a PD-PD configuration, achieving a three fold reduction of the double pendulum maximum swinging angle.This work has been partially supported by FEDER funds through MINECO project TIN2017-85827-P, and project KK-202000044 of the Elkartek 2020 funding program of the Basque Government. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 777720

Controlling a Quadrotor Carrying a Cable-Suspended Load to Pass Through a Window

In this paper, we design an optimal control system for a quadrotor to carry a cable-suspended load flying through a window. As the window is narrower than the length of the cable, it is very challenging to design a practical control system to pass through it. Our solution includes a system identification component, a trajectory generation component, and a trajectory tracking control component. The exact dynamic model that usually derived from the first principles is assumed to be unavailable. Instead, a model identification approach is adopted, which relies on a simple but effective low order equivalent system (LOES) to describe the core dynamical characteristics of the system. After being excited by some specifically designed manoeuvres, the unknown parameters in the LOES are obtained by using a frequency based least square estimation algorithm. Based on the estimated LOES, a numerical optimization algorithm is then utilized for aggressive trajectory generation when relevant constraints are given. The generated trajectory can lead to the quadrotor and load system passing through a narrow window with a cascade PD trajectory tracking controller. Finally, a practical flight test based on an Astec Hummingbird quadrotor is demonstrated and the result validates the proposed approach

A review of aerial manipulation of small-scale rotorcraft unmanned robotic systems

Small-scale rotorcraft unmanned robotic systems (SRURSs) are a kind of unmanned rotorcraft with manipulating devices. This review aims to provide an overview on aerial manipulation of SRURSs nowadays and promote relative research in the future. In the past decade, aerial manipulation of SRURSs has attracted the interest of researchers globally. This paper provides a literature review of the last 10 years (2008–2017) on SRURSs, and details achievements and challenges. Firstly, the definition, current state, development, classification, and challenges of SRURSs are introduced. Then, related papers are organized into two topical categories: mechanical structure design, and modeling and control. Following this, research groups involved in SRURS research and their major achievements are summarized and classified in the form of tables. The research groups are introduced in detail from seven parts. Finally, trends and challenges are compiled and presented to serve as a resource for researchers interested in aerial manipulation of SRURSs. The problem, trends, and challenges are described from three aspects. Conclusions of the paper are presented, and the future of SRURSs is discussed to enable further research interests

Human Demonstrations for Fast and Safe Exploration in Reinforcement Learning

<p>Reinforcement learning is a promising framework for controlling complex vehicles with a high level of autonomy, since it does not need a dynamic model of the vehicle, and it is able to adapt to changing conditions. When learning from scratch, the performance of a reinforcement learning controller may initially be poor and -for real life applications- unsafe. In this paper the effects of using human demonstrations on the performance of reinforcement learning is investigated, using a combination of offline and online least squares policy iteration. It is found that using the human as an efficient explorer improves learning time and performance for a benchmark reinforcement learning problem. The benefit of the human demonstration is larger for problems where the human can make use of its understanding of the problem to efficiently explore the state space. Applied to a simplified quadrotor slung load drop off problem, the use of human demonstrations reduces the number of crashes during learning. As such, this paper contributes to safer and faster learning for model-free, adaptive control problems.</p