Individual Human Behavior Identification Using an Inverse Reinforcement Learning Method

Shared control techniques have a great potential to create synergies in human-machine interaction for efficient and safe applications. However, an optimal interaction requires the machine to consider the individual behavior of the human partner. A widespread approach for modeling human behavior is given by optimal control theory, where the movement trajectories of a human arise from an optimized cost function. The aim of the identification is thus to determine parameters of a cost function which explains observed human motion. The central thesis of this paper is that individual cost function parameters which describe specific behavior can be determined by means of Inverse Reinforcement Learning. We show the applicability of the approach with a tracking control task example. The experiment consists in following a reference trajectory by means of a steering wheel. The study confirms that optimal control is suitable for modeling individual human behavior and demonstrates the suitability of Inverse Reinforcement Learning in order to determine the cost function parameters which explain measured data

Understanding robust control theory via stick balancing

Robust control theory studies the effect of noise, disturbances, and other uncertainty on system performance. Despite growing recognition across science and engineering that robustness and efficiency tradeoffs dominate the evolution and design of complex systems, the use of robust control theory remains limited, partly because the mathematics involved is relatively inaccessible to nonexperts, and the important concepts have been inexplicable without a fairly rich mathematics background. This paper aims to begin changing that by presenting the most essential concepts in robust control using human stick balancing, a simple case study popular in both the sensorimotor control literature and extremely familiar to engineers. With minimal and familiar models and mathematics, we can explore the impact of unstable poles and zeros, delays, and noise, which can then be easily verified with simple experiments using a standard extensible pointer. Despite its simplicity, this case study has extremes of robustness and fragility that are initially counter-intuitive but for which simple mathematics and experiments are clear and compelling. The theory used here has been well-known for many decades, and the cart-pendulum example is a standard in undergrad controls courses, yet a careful reconsidering of both leads to striking new insights that we argue are of great pedagogical value

A Bayesian perspective on classical control

The connections between optimal control and Bayesian inference have long been recognised, with the field of stochastic (optimal) control combining these frameworks for the solution of partially observable control problems. In particular, for the linear case with quadratic functions and Gaussian noise, stochastic control has shown remarkable results in different fields, including robotics, reinforcement learning and neuroscience, especially thanks to the established duality of estimation and control processes. Following this idea we recently introduced a formulation of PID control, one of the most popular methods from classical control, based on active inference, a theory with roots in variational Bayesian methods, and applications in the biological and neural sciences. In this work, we highlight the advantages of our previous formulation and introduce new and more general ways to tackle some existing problems in current controller design procedures. In particular, we consider 1) a gradient-based tuning rule for the parameters (or gains) of a PID controller, 2) an implementation of multiple degrees of freedom for independent responses to different types of signals (e.g., two-degree-of-freedom PID), and 3) a novel time-domain formalisation of the performance-robustness trade-off in terms of tunable constraints (i.e., priors in a Bayesian model) of a single cost functional, variational free energy.Comment: 8 pages, Accepted at IJCNN 202

Correlations in state space can cause sub-optimal adaptation of optimal feedback control models

Control of our movements is apparently facilitated by an adaptive internal model in the cerebellum. It was long thought that this internal model implemented an adaptive inverse model and generated motor commands, but recently many reject that idea in favor of a forward model hypothesis. In theory, the forward model predicts upcoming state during reaching movements so the motor cortex can generate appropriate motor commands. Recent computational models of this process rely on the optimal feedback control (OFC) framework of control theory. OFC is a powerful tool for describing motor control, it does not describe adaptation. Some assume that adaptation of the forward model alone could explain motor adaptation, but this is widely understood to be overly simplistic. However, an adaptive optimal controller is difficult to implement. A reasonable alternative is to allow forward model adaptation to ‘re-tune’ the controller. Our simulations show that, as expected, forward model adaptation alone does not produce optimal trajectories during reaching movements perturbed by force fields. However, they also show that re-optimizing the controller from the forward model can be sub-optimal. This is because, in a system with state correlations or redundancies, accurate prediction requires different information than optimal control. We find that adding noise to the movements that matches noise found in human data is enough to overcome this problem. However, since the state space for control of real movements is far more complex than in our simple simulations, the effects of correlations on re-adaptation of the controller from the forward model cannot be overlooked

Attention-dependent modulation of cortical taste circuits revealed by granger causality with signal-dependent noise

We show, for the first time, that in cortical areas, for example the insular, orbitofrontal, and lateral prefrontal cortex, there is signal-dependent noise in the fMRI blood-oxygen level dependent (BOLD) time series, with the variance of the noise increasing approximately linearly with the square of the signal. Classical Granger causal models are based on autoregressive models with time invariant covariance structure, and thus do not take this signal-dependent noise into account. To address this limitation, here we describe a Granger causal model with signal-dependent noise, and a novel, likelihood ratio test for causal inferences. We apply this approach to the data from an fMRI study to investigate the source of the top-down attentional control of taste intensity and taste pleasantness processing. The Granger causality with signal-dependent noise analysis reveals effects not identified by classical Granger causal analysis. In particular, there is a top-down effect from the posterior lateral prefrontal cortex to the insular taste cortex during attention to intensity but not to pleasantness, and there is a top-down effect from the anterior and posterior lateral prefrontal cortex to the orbitofrontal cortex during attention to pleasantness but not to intensity. In addition, there is stronger forward effective connectivity from the insular taste cortex to the orbitofrontal cortex during attention to pleasantness than during attention to intensity. These findings indicate the importance of explicitly modeling signal-dependent noise in functional neuroimaging, and reveal some of the processes involved in a biased activation theory of selective attention