FLEX: an Adaptive Exploration Algorithm for Nonlinear Systems

Model-based reinforcement learning is a powerful tool, but collecting data to fit an accurate model of the system can be costly. Exploring an unknown environment in a sample-efficient manner is hence of great importance. However, the complexity of dynamics and the computational limitations of real systems make this task challenging. In this work, we introduce FLEX, an exploration algorithm for nonlinear dynamics based on optimal experimental design. Our policy maximizes the information of the next step and results in an adaptive exploration algorithm, compatible with generic parametric learning models and requiring minimal resources. We test our method on a number of nonlinear environments covering different settings, including time-varying dynamics. Keeping in mind that exploration is intended to serve an exploitation objective, we also test our algorithm on downstream model-based classical control tasks and compare it to other state-of-the-art model-based and model-free approaches. The performance achieved by FLEX is competitive and its computational cost is low.Comment: Accepted at ICML 202

Redundant Hybrid Cable-Driven Robots: Modeling, Control, and Analysis

Serial and Cable-Driven Parallel Robots (CDPRs) are two types of robots that are widely used in industrial applications. Usually, the former offers high position accuracy at the cost of high motion inertia and small workspace envelope. The latter has a large workspace, low motion inertia, and high motion accelerations, but low accuracy. In this thesis, redundant Hybrid Cable-Driven Robots (HCDRs) are proposed to harness the strengths and benefits of serial and CDPRs. Although the study has been directed at warehousing applications, the developed techniques are general and can be applied to other applications. The main goal of this research is to develop integrated control systems to reduce vibrations and improve the position accuracy of HCDRs. For the proposed HCDRs, the research includes system modeling, redundancy resolution, optimization problem formulation, integrated control system development, and simulation and experimental validation. In this thesis, first, a generalized HCDR is proposed for the step-by-step derivation of a generic model, and it can be easily extended to any HCDRs. Then, based on an in-plane configuration, three types of control architecture are proposed to reduce vibrations and improve the position accuracy of HCDR. Their performance is evaluated using several well-designed case studies. Furthermore, a stiffness optimization algorithm is developed to overcome the limitations of existing approaches. Decoupled system modeling is studied to reduce the complexity of HCDRs. Control design, simulations, and experiments are developed to validate the models and control strategies. Additionally, state estimation algorithms are proposed to overcome the inaccurate limitation of Inertial Measurement Unit (IMU). Based on these state observers, experiments are conducted in different cases to evaluate the control performance. An Underactuated Mobile Manipulator (UMM) is proposed to address the tracking and vibration- and balance-control problems. Out-of-plane system modeling, disturbance analysis, and model validation are also investigated. Besides, a simple but effective strategy is developed to solve the equilibrium point and balancing problem. Based on the dynamic model, two control architectures are proposed. Compared to other Model Predictive Control (MPC)-based control strategies, the proposed controllers require less effort to implement in practice. Simulations and experiments are also conducted to evaluate the model and control performance. Finally, redundancy resolution and disturbance rejection via torque optimization in HCDRs are proposed: joint-space Torque Optimization for Actuated Joints (TOAJ) and joint-space Torque Optimization for Actuated and Unactuated Joints (TOAUJ). Compared to TOAJ, TOAUJ can solve the redundancy resolution problem as well as disturbance rejection. The algorithms are evaluated using a Three-Dimensional (3D) coupled HCDR and can also be extended to other HCDRs

A Practical and Conceptual Framework for Learning in Control

We propose a fully Bayesian approach for efficient reinforcement learning (RL) in Markov decision processes with continuous-valued state and action spaces when no expert knowledge is available. Our framework is based on well-established ideas from statistics and machine learning and learns fast since it carefully models, quantifies, and incorporates available knowledge when making decisions. The key ingredient of our framework is a probabilistic model, which is implemented using a Gaussian process (GP), a distribution over functions. In the context of dynamic systems, the GP models the transition function. By considering all plausible transition functions simultaneously, we reduce model bias, a problem that frequently occurs when deterministic models are used. Due to its generality and efficiency, our RL framework can be considered a conceptual and practical approach to learning models and controllers whe

A kinematic controller for liquid pouring between vessels modelled with smoothed particle hydrodynamics

In robotics, the task of pouring liquids into vessels in non-structured or domestic spaces is an open field of study. A real time, fluid dynamic simulation, based on smoothed particle hydrodynamics (SPH), together with solid motion kinematics, allow for a closed loop control of pouring. In the first place, a control criterion related with the behavior of the liquid free surface is established to handle sloshing, especially in the initial phase of pouring to prevent liquid adhesion over the vessel rim. A 2-D, free surface SPH simulation is implemented on a graphic processing unit (GPU) to predict the liquid motion with real-time capability. The pouring vessel has a single degree of freedom of rotation, while the catching vessel has a single degree of freedom of translation, and the control loop handles the tilting angle of the pouring vessel. In this work, a two-stage pouring method is proposed, differentiating an initial phase where sloshing is particularly relevant, and a nearly constant outflow phase. For control purposes, the free outflow trajectory was simplified and modelled as a free falling solid with an initial velocity at the vessel crest, as calculated by the SPH simulation. As the first stage of pouring is more delicate, a novel slosh induction method (SIM) is proposed to overcome spilling issues during initial tilting in full filled vessels. Both robotic control and fluid modelling showed good results at multiples initial vessel filling heights

Concurrent design and motion planning in robotics using differentiable optimal control

Robot design optimization (what the robot is) and motion planning (how the robot moves) are two problems that are connected. Robots are limited by their design in terms of what motions they can execute – for instance a robot with a heavy base has less payload capacity compared to the same robot with a lighter base. On the other hand, the motions that the robot executes guide which design is best for the task. Concurrent design (co-design) is the process of performing robot design and motion planning together. Although traditionally co-design has been viewed as an offline process that can take hours or days, we view interactive co-design tools as the next step as they enable quick prototyping and evaluation of designs across different tasks and environments. In this thesis we adopt a gradient-based approach to co-design. Our baseline approach embeds the motion planning into bi-level optimization and uses gradient information via finite differences from the lower motion planning level to optimize the design in the upper level. Our approach uses the full rigid-body dynamics of the robot and allows for arbitrary upper-level design constraints, which is key for finding physically realizable designs. Our approach is also between 1.8 and 8.4 times faster on a quadruped trotting and jumping co-design task as compared to the popular genetic algorithm covariance matrix adaptation evolutionary strategy (CMA-ES). We further demonstrate the speed of our approach by building an interactive co-design tool that allows for optimization over uneven terrain with varying height. Furthermore, we propose an algorithm to analytically take the derivative of nonlinear optimal control problems via differential dynamic programming (DDP). Analytical derivatives are a step towards addressing the scalability and accuracy issues of finite differences. We further compared with a simultaneous approach for co-design that optimizes both motion and design in one nonlinear program. On a co-design task for the Kinova robotic arm we observed a 54-times improvement in computational speed. We additionally carry out hardware validation experiments on the quadruped robot Solo. We designed longer lower legs for the robot, which minimize the peak torque used during trotting. Although we always observed an improvement in peak torque, it was less than in simulation (7.609% versus 28.271%). We discuss some of the sim-toreal issues including the structural stability of joints and slipping of feet that need to be considered and how they can be addressed using our framework. In the second part of this thesis we propose solutions to some open problems in motion planning. Firstly, in our co-design approach we assumed fixed contact locations and timings. Ideally we would like the motion planner to choose the contacts instead. We solve a related, but simpler problem, which is the control of satellite thrusters, which are similar to robot feet but do not have the constraint of having to be in contact with the ground to exert force on the robot. We introduce a sparse, L1 cost on control inputs (thrusters) and implement optimization via DDP-style solvers. We use full rigid-body dynamics and achieve bang-bang control via optimization, which is a difficult problem due to the discrete switching nature of the thrusters. Lastly, we present a method for planning and control of a hybrid, wheel-legged robot. This is a difficult problem, as the robot needs to always actively balance on the wheel even when not driving or jumping forward. We propose the variablelength wheeled inverted pendulum (VL-WIP) template model that captures only the necessary dynamic interactions between wheels and base. We embedded this into a model-predictive controller (MPC) and demonstrated highly dynamic behaviors, including swinging-up and jumping over a gap. Both of these motion planning problems expand the ability of our motion planning tools to new domains, which is an integral part also of the co-design algorithms, as co-design aims to optimize both design, and motion, together

Rotating potential of a stochastic parametric pendulum

The parametric pendulum is a fruitful dynamical system manifesting some of the most interesting phenomena of nonlinear dynamics, well-known to exhibit rather complex motion including period doubling, fold and pitchfork bifurcations, let alone the global bifurcations leading to chaotic or rotational motion. In this thesis, the potential of establishing rotational motion is studied considering the bobbing motion of ocean waves as the source of excitation of a oating pendulum. The challenge within this investigation lies on the fact that waves are random, as well as their observed low frequency, characteristics which pose a broader signi cance within the study of vibrating systems. Thus, a generic study is conducted with the parametric pendulum being excited by a narrow-band stochastic process and particularly, the random phase modulation is utilized. In order to explore the dynamics of the stochastic system, Markov-chain Monte-Calro simulations are performed to acquire a view on the in uence of randomness onto the parameter regions leading to rotational response. Furthermore, the Probability Density Function of the response is calculated, applying a numerical iterative scheme to solve the total probability law, exploiting the Chapman-Kolmogorov equation inherent to Markov processes. A special case of the studied structure undergoing impacts is considered to account for extreme weather conditions and nally, a novel design is investigated experimentally, aiming to set the ground for future development