Composable Deep Reinforcement Learning for Robotic Manipulation

Model-free deep reinforcement learning has been shown to exhibit good performance in domains ranging from video games to simulated robotic manipulation and locomotion. However, model-free methods are known to perform poorly when the interaction time with the environment is limited, as is the case for most real-world robotic tasks. In this paper, we study how maximum entropy policies trained using soft Q-learning can be applied to real-world robotic manipulation. The application of this method to real-world manipulation is facilitated by two important features of soft Q-learning. First, soft Q-learning can learn multimodal exploration strategies by learning policies represented by expressive energy-based models. Second, we show that policies learned with soft Q-learning can be composed to create new policies, and that the optimality of the resulting policy can be bounded in terms of the divergence between the composed policies. This compositionality provides an especially valuable tool for real-world manipulation, where constructing new policies by composing existing skills can provide a large gain in efficiency over training from scratch. Our experimental evaluation demonstrates that soft Q-learning is substantially more sample efficient than prior model-free deep reinforcement learning methods, and that compositionality can be performed for both simulated and real-world tasks.Comment: Videos: https://sites.google.com/view/composing-real-world-policies

Fuzzy robust nonlinear control approach for electro-hydraulic flight motion simulator

AbstractA fuzzy robust nonlinear controller for hydraulic rotary actuators in flight motion simulators is proposed. Compared with other three-order models of hydraulic rotary actuators, the proposed controller based on first-order nonlinear model is more easily applied in practice, whose control law is relatively simple. It not only does not need high-order derivative of desired command, but also does not require the feedback signals of velocity, acceleration and jerk of hydraulic rotary actuators. Another advantage is that it does not rely on any information of friction, inertia force and external disturbing force/torque, which are always difficult to resolve in flight motion simulators. Due to the special composite vane seals of rectangular cross-section and goalpost shape used in hydraulic rotary actuators, the leakage model is more complicated than that of traditional linear hydraulic cylinders. Adaptive multi-input single-output (MISO) fuzzy compensators are introduced to estimate nonlinear uncertain functions about leakage and bulk modulus. Meanwhile, the decomposition of the uncertainties is used to reduce the total number of fuzzy rules. Different from other adaptive fuzzy compensators, a discontinuous projection mapping is employed to guarantee the estimation process to be bounded. Furthermore, with a sufficient number of fuzzy rules, the controller theoretically can guarantee asymptotic tracking performance in the presence of the above uncertainties, which is very important for high-accuracy tracking control of flight motion simulators. Comparative experimental results demonstrate the effectiveness of the proposed algorithm, which can guarantee transient performance and better final accurate tracking in the presence of uncertain nonlinearities and parametric uncertainties