Intrinsic Motivation and Mental Replay enable Efficient Online Adaptation in Stochastic Recurrent Networks

Autonomous robots need to interact with unknown, unstructured and changing environments, constantly facing novel challenges. Therefore, continuous online adaptation for lifelong-learning and the need of sample-efficient mechanisms to adapt to changes in the environment, the constraints, the tasks, or the robot itself are crucial. In this work, we propose a novel framework for probabilistic online motion planning with online adaptation based on a bio-inspired stochastic recurrent neural network. By using learning signals which mimic the intrinsic motivation signalcognitive dissonance in addition with a mental replay strategy to intensify experiences, the stochastic recurrent network can learn from few physical interactions and adapts to novel environments in seconds. We evaluate our online planning and adaptation framework on an anthropomorphic KUKA LWR arm. The rapid online adaptation is shown by learning unknown workspace constraints sample-efficiently from few physical interactions while following given way points.Comment: accepted in Neural Network

Extremum Seeking-based Iterative Learning Linear MPC

In this work we study the problem of adaptive MPC for linear time-invariant uncertain models. We assume linear models with parametric uncertainties, and propose an iterative multi-variable extremum seeking (MES)-based learning MPC algorithm to learn on-line the uncertain parameters and update the MPC model. We show the effectiveness of this algorithm on a DC servo motor control example.Comment: To appear at the IEEE MSC 201

Batch-to-batch iterative learning control of a fed-batch fermentation process

PhD ThesisRecently, iterative learning control (ILC) has been used in the run-to-run control of batch processes to directly update the control trajectory. The basic idea of ILC is to update the control trajectory for a new batch run using the information from previous batch runs so that the output trajectory converges asymptotically to the desired reference trajectory. The control policy updating is calculated using linearised models around the nominal reference process input and output trajectories. The linearised models are typically identified using multiple linear regression (MLR), partial least squares (PLS) regression, or principal component regression (PCR). ILC has been shown to be a promising method to address model-plant mismatches and unknown disturbances. This work presents several improvements of batch to batch ILC strategy with applications to a simulated fed-batch fermentation process. In order to enhance the reliability of ILC, model prediction confidence is incorporated in the ILC optimization objective function. As a result of the incorporation, wide model prediction confidence bounds are penalized in order to avoid unreliable control policy updating. This method has been proven to be very effective for selected model prediction confidence bounds penalty factors. In the attempt to further improve the performance of ILC, averaged reference trajectories and sliding window techniques were introduced. To reduce the influence of measurement noise, control policy is updated on the average input and output trajectories of the past a few batches instead of just the immediate previous batch. The linearised models are re-identified using a sliding window of past batches in that the earliest batch is removed with the newest batch added to the model identification data set. The effects of various parameters were investigated for MLR, PCR and PLS method. The technique significantly improves the control performance. In model based ILC the weighting matrices, Q and R, in the objective function have a significant impact on the control performance. Therefore, in the quest to exploit the potential of objective function, adaptive weighting parameters were attempted to study the performance of batch to batch ILC with updated models. Significant improvements in the stability of the performance for all the three methods were noticed. All the three techniques suggested have established improvements either in stability, reliability and/or convergence speed. To further investigate the versatility of ILC, the above mentioned techniques were combined and the results are discussed in this thesis

Robust PID based indirect-type iterative learning control for batch processes with time-varying uncertainties

ased on the proportional-integral-derivative (PID) control structure widely used in engineering applications, a robust indirect-type iterative learning control (ILC) method is proposed for industrial batch processes subject to time-varying uncertainties. An important merit is that the proposed ILC design is independent of the PID tuning that aims primarily to hold robust stability of the closed-loop system, owing to the fact that the ILC updating law is implemented through adjusting the setpoint of the closed-loop PID control structure plus a feedforward control to the plant input from batch to batch. According to the robust H infinity control objective, a robust discrete-time PID tuning algorithm is given in terms of the plant state-space model description to accommodate for time-varying process uncertainties. For the batchwise direction, a robust ILC updating law is developed based on the two-dimensional (2D) control system theory. Only measured output errors of current and previous cycles are used to implement the proposed ILC scheme for the convenience of practical application. An illustrative example from the literature is adopted to demonstrate the effectiveness and merits of the proposed ILC method

Model learning for trajectory tracking of robot manipulators

Abstract Model based controllers have drastically improved robot performance, increasing task accuracy while reducing control effort. Nevertheless, all this was realized with a very strong assumption: the exact knowledge of the physical properties of both the robot and the environment that surrounds it. This assertion is often misleading: in fact modern robots are modeled in a very approximate way and, more important, the environment is almost never static and completely known. Also for systems very simple, such as robot manipulators, these assumptions are still too strong and must be relaxed. Many methods were developed which, exploiting previous experiences, are able to refine the nominal model: from classic identification techniques to more modern machine learning based approaches. Indeed, the topic of this thesis is the investigation of these data driven techniques in the context of robot control for trajectory tracking. In the first two chapters, preliminary knowledge is provided on both model based controllers, used in robotics to assure precise trajectory tracking, and model learning techniques. In the following three chapters, are presented the novelties introduced by the author in this context with respect to the state of the art: three works with the same premise (an inaccurate system modeling), an identical goal (accurate trajectory tracking control) but with small differences according to the specific platform of application (fully actuated, underactuated, redundant robots). In all the considered architectures, an online learning scheme has been introduced to correct the nominal feedback linearization control law. Indeed, the method has been primarily introduced in the literature to cope with fully actuated systems, showing its efficacy in the accurate tracking of joint space trajectories also with an inaccurate dynamic model. The main novelty of the technique was the use of only kinematics information, instead of torque measurements (in general very noisy), to online retrieve and compensate the dynamic mismatches. After that the method has been extended to underactuated robots. This new architecture was composed by an online learning correction of the controller, acting on the actuated part of the system (the nominal partial feedback linearization), and an offline planning phase, required to realize a dynamically feasible trajectory also for the zero dynamics of the system. The scheme was iterative: after each trial, according to the collected information, both the phases were improved and then repeated until the task achievement. Also in this case the method showed its capability, both in numerical simulations and on real experiments on a robotics platform. Eventually the method has been applied to redundant systems: differently from before, in this context the task consisted in the accurate tracking of a Cartesian end effector trajectory. In principle very similar to the fully actuated case, the presence of redundancy slowed down drastically the learning machinery convergence, worsening the performance. In order to cope with this, a redundancy resolution was proposed that, exploiting an approximation of the learning algorithm (Gaussian process regression), allowed to locally maximize the information and so select the most convenient self motion for the system; moreover, all of this was realized with just the resolution of a quadratic programming problem. Also in this case the method showed its performance, realizing an accurate online tracking while reducing both the control effort and the joints velocity, obtaining so a natural behaviour. The thesis concludes with summary considerations on the proposed approach and with possible future directions of research

Iterative learning control of crystallisation systems

Under the increasing pressure of issues like reducing the time to market, managing lower production costs, and improving the flexibility of operation, batch process industries thrive towards the production of high value added commodity, i.e. specialty chemicals, pharmaceuticals, agricultural, and biotechnology enabled products. For better design, consistent operation and improved control of batch chemical processes one cannot ignore the sensing and computational blessings provided by modern sensors, computers, algorithms, and software. In addition, there is a growing demand for modelling and control tools based on process operating data. This study is focused on developing process operation data-based iterative learning control (ILC) strategies for batch processes, more specifically for batch crystallisation systems. In order to proceed, the research took a step backward to explore the existing control strategies, fundamentals, mechanisms, and various process analytical technology (PAT) tools used in batch crystallisation control. From the basics of the background study, an operating data-driven ILC approach was developed to improve the product quality from batch-to-batch. The concept of ILC is to exploit the repetitive nature of batch processes to automate recipe updating using process knowledge obtained from previous runs. The methodology stated here was based on the linear time varying (LTV) perturbation model in an ILC framework to provide a convergent batch-to-batch improvement of the process performance indicator. In an attempt to create uniqueness in the research, a novel hierarchical ILC (HILC) scheme was proposed for the systematic design of the supersaturation control (SSC) of a seeded batch cooling crystalliser. This model free control approach is implemented in a hierarchical structure by assigning data-driven supersaturation controller on the upper level and a simple temperature controller in the lower level. In order to familiarise with other data based control of crystallisation processes, the study rehearsed the existing direct nucleation control (DNC) approach. However, this part was more committed to perform a detailed strategic investigation of different possible structures of DNC and to compare the results with that of a first principle model based optimisation for the very first time. The DNC results in fact outperformed the model based optimisation approach and established an ultimate guideline to select the preferable DNC structure. Batch chemical processes are distributed as well as nonlinear in nature which need to be operated over a wide range of operating conditions and often near the boundary of the admissible region. As the linear lumped model predictive controllers (MPCs) often subject to severe performance limitations, there is a growing demand of simple data driven nonlinear control strategy to control batch crystallisers that will consider the spatio-temporal aspects. In this study, an operating data-driven polynomial chaos expansion (PCE) based nonlinear surrogate modelling and optimisation strategy was presented for batch crystallisation processes. Model validation and optimisation results confirmed this approach as a promise to nonlinear control. The evaluations of the proposed data based methodologies were carried out by simulation case studies, laboratory experiments and industrial pilot plant experiments. For all the simulation case studies a detailed mathematical models covering reaction kinetics and heat mass balances were developed for a batch cooling crystallisation system of Paracetamol in water. Based on these models, rigorous simulation programs were developed in MATLAB®, which was then treated as the real batch cooling crystallisation system. The laboratory experimental works were carried out using a lab scale system of Paracetamol and iso-Propyl alcohol (IPA). All the experimental works including the qualitative and quantitative monitoring of the crystallisation experiments and products demonstrated an inclusive application of various in situ process analytical technology (PAT) tools, such as focused beam reflectance measurement (FBRM), UV/Vis spectroscopy and particle vision measurement (PVM) as well. The industrial pilot scale study was carried out in GlaxoSmithKline Bangladesh Limited, Bangladesh, and the system of experiments was Paracetamol and other powdered excipients used to make paracetamol tablets. The methodologies presented in this thesis provide a comprehensive framework for data-based dynamic optimisation and control of crystallisation processes. All the simulation and experimental evaluations of the proposed approaches emphasised the potential of the data-driven techniques to provide considerable advances in the current state-of-the-art in crystallisation control