Proximal Policy Optimization for Tracking Control Exploiting Future Reference Information

In recent years, reinforcement learning (RL) has gained increasing attention in control engineering. Especially, policy gradient methods are widely used. In this work, we improve the tracking performance of proximal policy optimization (PPO) for arbitrary reference signals by incorporating information about future reference values. Two variants of extending the argument of the actor and the critic taking future reference values into account are presented. In the first variant, global future reference values are added to the argument. For the second variant, a novel kind of residual space with future reference values applicable to model-free reinforcement learning is introduced. Our approach is evaluated against a PI controller on a simple drive train model. We expect our method to generalize to arbitrary references better than previous approaches, pointing towards the applicability of RL to control real systems

Metoder för rörelseplanering och analys av underaktuerade mekaniska system och redundanta manipulatorer

Motion planning and control synthesis are challenging problems for underactuated mechanical systems due to the presence of passive (non-actuated) degrees of freedom. For those systems that are additionally not feedback linearizable and with unstable internal dynamics there are no generic methods for planning trajectories and their feedback stabilization. For fully actuated mechanical systems, on the other hand, there are standard tools that provide a tractable solution. Still, the problem of generating efficient and optimal trajectories is nontrivial due to actuator limitations and motion-dependent velocity and acceleration constraints that are typically present. It is especially challenging for manipulators with kinematic redundancy. A generic approach for solving the above-mentioned problems is described in this work. We explicitly use the geometry of the state space of the mechanical system so that a synchronization of the generalized coordinates can be found in terms of geometric relations along the target motion with respect to a path coordinate. Hence, the time evolution of the state variables that corresponds to the target motion is determined by the system dynamics constrained to these geometrical relations, known as virtual holonomic constraints. Following such a reduction for underactuated mechanical systems, we arrive at integrable second-order dynamics associated with the passive degrees of freedom. Solutions of this reduced dynamics, together with the geometric relations, can be interpreted as a motion generator for the full system. For fully actuated mechanical systems the virtually constrained dynamics provides a tractable way of shaping admissible trajectories. Once a feasible target motion is found and the corresponding virtual holonomic constraints are known, we can describe dynamics transversal to the orbit in the state space and analytically compute a transverse linearization. This results in a linear time-varying control system that allows us to use linear control theory for achieving orbital stabilization of the nonlinear mechanical system as well as to conduct system analysis in the vicinity of the motion. The approach is applicable to continuous-time and impulsive mechanical systems irrespective of the degree of underactuation. The main contributions of this thesis are analysis of human movement regarding a nominal behavior for repetitive tasks, gait synthesis and stabilization for dynamic walking robots, and description of a numerical procedure for generating and stabilizing efficient trajectories for kinematically redundant manipulators

Applications of the Virtual Holonomic Constraints Approach : Analysis of Human Motor Patterns and Passive Walking Gaits

In the field of robotics there is a great interest in developing strategies and algorithms to reproduce human-like behavior. One can think of human-like machines that may replace humans in hazardous working areas, perform enduring assembly tasks, serve the elderly and handicapped, etc. The main challenges in the development of such robots are, first, to construct sophisticated electro-mechanical humanoids and, second, to plan and control human-like motor patterns. A promising idea for motion planning and control is to reparameterize any somewhat coordinated motion in terms of virtual holonomic constraints, i.e. trajectories of all degrees of freedom of the mechanical system are described by geometric relations among the generalized coordinates. Imposing such virtual holonomic constraints on the system dynamics allows to generate synchronized motor patterns by feedback control. In fact, there exist consistent geometric relations in ordinary human movements that can be used advantageously. In this thesis the virtual constraints approach is extended to a wider and rigorous use for analyzing, planning and reproducing human-like motions based on mathematical tools previously utilized for very particular control problems. It is often the case that some desired motions cannot be achieved by the robot due to limitations in available actuation power. This constraint rises the question of how to modify the mechanical design in order to achieve better performance. An underactuated planar two-link robot is used to demonstrate that springs can complement the actuation in parallel to an ordinary motor. Motion planning is carried out for the original robot dynamics while the springs are treated as part of the control action with a torque profile suited to the preplanned trajectory. Another issue discussed in this thesis is to find stable and unstable (hybrid) limit cycles for passive dynamic walking robots without integrating the full set of differential equations. Such procedure is demonstrated for the compass-gait biped by means of optimization with a reduced number of initial conditions and parameters to search. The properties of virtual constraints and reduced dynamics are exploited to solve this problem

Motion planning and control of an underactuated 3DOF helicopter

We consider trajectory planning for an underactuated 3DOF helicopter, using the virtual holonomic constraint approach. First we choose constraint functions that describe the configuration variables along a desired motion in terms of some independent parametrization variable. This lets us describe the closed-loop system by some reduced order dynamics, the solution of which gives a feasible trajectory for the desired motion. By using the method of transverse linearization for controller design, we achieve exponential orbital stability to a desired trajectory. Numerical simulations confirm this property and show good convergence to a desired periodic motion when initialized from a resting state

Ball dribbling with an underactuated continuous-time control phase

Abstract — Ball dribbling is a central element of basketball. One main challenge for realizing basketball robots is to stabilize periodic motions of the ball. This task is nontrivial due to the discrete-continuous nature of the corresponding dynamics. The ball can be only controlled during ball-manipulator contact and moves freely otherwise. We propose a manipulator equipped with a spring that gets compressed when the ball bounces against it. Hence, we can have continuous-time control over this underactuated Ball-Spring-Manipulator system until the spring releases its accumulated energy back to the ball. This paper illustrates a systematic way of planning such a modified dribbling motion, computing an analytical transverse linearization and achieving orbital stabilization

Stable Dynamic Walking over Rough Terrain

We propose a constructive control design for stabilization of non-periodic trajectories of underactuated mechanical systems. An important example of such a system is an underactuated "dynamic walking" biped robot walking over rough terrain. The proposed technique is to compute a transverse linearization about the desired motion: a linear impulsive system which locally represents dynamics about a target trajectory. This system is then exponentially stabilized using a modified receding-horizon control design. The proposed method is experimentally verified using a compass-gait walker: a two-degree-of-freedom biped with hip actuation but pointed stilt-like feet. The technique is, however, very general and can be applied to higher degree-of-freedom robots over arbitrary terrain and other impulsive mechanical systems. © 2011 Springer-Verlag