A Survey on Policy Search for Robotics

Policy search is a subfield in reinforcement learning which focuses on finding good parameters for a given policy parametrization. It is well suited for robotics as it can cope with high-dimensional state and action spaces, one of the main challenges in robot learning. We review recent successes of both model-free and model-based policy search in robot learning. Model-free policy search is a general approach to learn policies based on sampled trajectories. We classify model-free methods based on their policy evaluation strategy, policy update strategy, and exploration strategy and present a unified view on existing algorithms. Learning a policy is often easier than learning an accurate forward model, and, hence, model-free methods are more frequently used in practice. However, for each sampled trajectory, it is necessary to interact with the * Both authors contributed equally. robot, which can be time consuming and challenging in practice. Modelbased policy search addresses this problem by first learning a simulator of the robot’s dynamics from data. Subsequently, the simulator generates trajectories that are used for policy learning. For both modelfree and model-based policy search methods, we review their respective properties and their applicability to robotic systems

Design and control of next-generation uavs for effectively interacting with environments

In this dissertation, the design and control of a novel multirotor for aerial manipulation is studied, with the aim of endowing the aerial vehicle with more degrees of freedom of motion and stability when interacting with the environments. Firstly, it presents an energy-efficient adaptive robust tracking control method for a class of fully actuated, thrust vectoring unmanned aerial vehicles (UAVs) with parametric uncertainties including unknown moment of inertia, mass and center of mass, which would occur in aerial maneuvering and manipulation. The effectiveness of this method is demonstrated through simulation. Secondly, a humanoid robot arm is adopted to serve as a 6-degree-of-freedom (DOF) automated flight testing platform for emulating the free flight environment of UAVs while ensuring safety. Another novel multirotor in a tilt-rotor architecture is studied and tested for coping with parametric uncertainties in aerial maneuvering and manipulation. Two pairs of rotors are mounted on two independently-controlled tilting arms placed at two sides of the vehicle in a H configuration to enhance its maneuverability and stability through an adaptive robust control method. In addition, an impedance control algorithm is deployed in the out loop that modifies the trajectory to achieve a compliant behavior in the end-effector space for aerial drilling and screwing tasks

Control of Nonlinear Mechatronic Systems

This dissertation is divided into four self-contained chapters. In Chapter 1, an adaptive nonlinear tracking controller for kinematically redundant robot manipulators is presented. Past research efforts have focused on the end-effector tracking control of redundant robots because of their increased dexterity over their non-redundant counterparts. This work utilizes an adaptive full-state feedback quaternion based controller developed in [1] and focuses on the design of a general sub-task controller. This sub-task controller does not affect the position and orientation tracking control objectives, but instead projects a preference on the configuration of the manipulator based on sub-task objectives such as the following: singularity avoidance, joint limit avoidance, bounding the impact forces, and bounding the potential energy. In Chapter 2, two controllers are developed for nonlinear haptic and teleoperator systems for coordination of the master and slave systems. The first controller is proven to yield a semi-global asymptotic result in the presence of parametric uncertainty in the master and the slave dynamic models provided the user and the environmental input forces are measurable. The second controller yields a global asymptotic result despite unmeasurable user and environmental input forces provided the dynamic models of the master and slave systems are known. These controllers rely on a transformation and a flexible target system to allow the master system\u27s impedance to be easily adjusted so that it matches a desired target system. This work also offers a structure to encode a velocity field assist mechanism to provide the user help in controlling the slave system in completing a pre-defined contour following task. For each controller, Lyapunov-based techniques are used to prove that both controllers provide passive coordination of the haptic/teleoperator system when the velocity field assist mechanism is disabled. When the velocity field assist mechanism is enabled, the analysis proves the coordination of the haptic/teleoperator system. Simulation results are presented for both controllers. In Chapter 3, two controllers are developed for flat multi-input/multi-output nonlinear systems. First, a robust adaptive controller is proposed and proven to yield semi-global asymptotic tracking in the presence of additive disturbances and parametric uncertainty. In addition to guaranteeing an asymptotic output tracking result, it is also proven that the parameter estimate vector is driven to a constant vector. In the second part of the chapter, a learning controller is designed and proven to yield a semi-global asymptotic tracking result in the presence of additive disturbances where the desired trajectory is periodic. A continuous nonlinear integral feedback component is utilized in the design of both controllers and Lyapunov-based techniques are used to guarantee that the tracking error is asymptotically driven to zero. Numerical simulation results are presented for both controllers. In Chapter 4, a new dynamic model for continuum robot manipulators is derived. The dynamic model is developed based on the geometric model of extensible continuum robot manipulators with no torsional effects. The development presented in this chapter is an extension of the dynamic model proposed in [2] (by Mochiyama and Suzuki) to include a class of extensible continuum robot manipulators. First, the kinetic energy of a slice of the continuum robot is evaluated. Next, the total kinetic energy of the manipulator is obtained by utilizing a limit operation (i.e., sum of the kinetic energy of all the slices). Then, the gravitational potential energy of the manipulator is derived. Next, the elastic potential energy of the manipulator is derived for both bending and extension. Finally, the dynamic model of a planar 3-section extensible continuum robot manipulator is derived by utilizing the Lagrange representation. Numerical simulation results are presented for a planar 3-section extensible continuum robot manipulator

Adaptive Modified RISE-based Quadrotor Trajectory Tracking with Actuator Uncertainty Compensation

This paper presents an adaptive robust nonlinear control method, which achieves reliable trajectory tracking control for a quadrotor unmanned aerial vehicle in the presence of gyroscopic effects, rotor dynamics, and external disturbances. Through novel mathematical manipulation in the error system development, the quadrotor dynamics are expressed in a control-oriented form, which explicitly incorporates the uncertainty in the gyroscopic term and control actuation term. An adaptive robust nonlinear control law is then designed to stabilize both the position and attitude loops of the quadrotor system. A rigorous Lyapunov-based analysis is utilized to prove asymptotic trajectory tracking, where the region of convergence can be made arbitrarily large through judicious control gain selection. Moreover, the stability analysis formally addresses gyroscopic effects and actuator uncertainty. To illustrate the performance of the control law, comparative numerical simulation results are provided, which demonstrate the improved closed-loop performance achieved under varying levels of parametric uncertainty and disturbance magnitudes

Robust Image-Based Visual Servo Control of an Uncertain Missile Airframe

A nonlinear vision-based guidance law is presented for a missile-target scenario in the presence of model uncertainty and unknown target evasive maneuvers. To ease the readability of this thesis, detailed explanations of any relevant mathematical tools are provided, including stability definitions, the procedure of Lyapunov-based stability analysis, sliding mode control fundamentals, basics on visual servo control, and other basic nonlinear control tools. To develop the vision-based guidance law, projective geometric relationships are utilized to combine the image kinematics with the missile dynamics in an integrated visual dynamic system. The guidance law is designed using an image-based visual servo control method in conjunction with a sliding-mode control strategy, which is shown to achieve asymptotic target interception in the presence of the aforementioned uncertainties. A Lyapunov-based stability analysis is presented to prove the theoretical result, and numerical simulation results are provided to demonstrate the performance of the proposed robust controller for both stationary and non-stationary targets