A Review of Control Strategies in Closed-Loop Neuroprosthetic Systems

It has been widely recognized that closed-loop neuroprosthetic systems achieve more favourable outcomes for users then equivalent open-loop devices. Improved performance of tasks, better usability and greater embodiment have all been reported in systems utilizing some form of feedback. However the interdisciplinary work on neuroprosthetic systems can lead to miscommunication due to similarities in well established nomenclature in different fields. Here we present a review of control strategies in existing experimental, investigational and clinical neuroprosthetic systems in order to establish a baseline and promote a common understanding of different feedback modes and closed loop controllers. The first section provides a brief discussion of feedback control and control theory. The second section reviews the control strategies of recent Brain Machine Interfaces, neuromodulatory implants, neuroprosthetic systems and assistive neurorobotic devices. The final section examines the different approaches to feedback in current neuroprosthetic and neurorobotic systems

Learning Control Policies of Hodgkin-Huxley Neuronal Dynamics

We present a neural network approach for closed-loop deep brain stimulation (DBS). We cast the problem of finding an optimal neurostimulation strategy as a control problem. In this setting, control policies aim to optimize therapeutic outcomes by tailoring the parameters of a DBS system, typically via electrical stimulation, in real time based on the patient's ongoing neuronal activity. We approximate the value function offline using a neural network to enable generating controls (stimuli) in real time via the feedback form. The neuronal activity is characterized by a nonlinear, stiff system of differential equations as dictated by the Hodgkin-Huxley model. Our training process leverages the relationship between Pontryagin's maximum principle and Hamilton-Jacobi-Bellman equations to update the value function estimates simultaneously. Our numerical experiments illustrate the accuracy of our approach for out-of-distribution samples and the robustness to moderate shocks and disturbances in the system.Comment: Extended Abstract presented at Machine Learning for Health (ML4H) symposium 2023, December 10th, 2023, New Orleans, United States, 12 page

Personalized rTMS for Depression: A Review

Personalized treatments are gaining momentum across all fields of medicine. Precision medicine can be applied to neuromodulatory techniques, where focused brain stimulation treatments such as repetitive transcranial magnetic stimulation (rTMS) are used to modulate brain circuits and alleviate clinical symptoms. rTMS is well-tolerated and clinically effective for treatment-resistant depression (TRD) and other neuropsychiatric disorders. However, despite its wide stimulation parameter space (location, angle, pattern, frequency, and intensity can be adjusted), rTMS is currently applied in a one-size-fits-all manner, potentially contributing to its suboptimal clinical response (~50%). In this review, we examine components of rTMS that can be optimized to account for inter-individual variability in neural function and anatomy. We discuss current treatment options for TRD, the neural mechanisms thought to underlie treatment, differences in FDA-cleared devices, targeting strategies, stimulation parameter selection, and adaptive closed-loop rTMS to improve treatment outcomes. We suggest that better understanding of the wide and modifiable parameter space of rTMS will greatly improve clinical outcome

Optimizing electrical brain stimulation for seizure disorders

University of Minnesota Ph.D. dissertation. March 2017. Major: Neuroscience. Advisor: Theoden Netoff. 1 computer file (PDF); x, 145 pages.Approximately 1% of the world population is afflicted with Epilepsy. For many patients, antiepileptic drugs do not fully control seizures. Electrical brain stimulation therapies have been effective in reducing seizure rates in some patients. While current neuromodulation devices provide a benefit to patients, efficacy can be improved by optimizing brain stimulation so that the therapy is tuned on a patient by patient basis. One optimization approach is to target deep brain regions that strongly modulate seizure prone regions. I will present data on the effects of stimulation of two different anatomical regions for seizure control, and establish my experimental platform for testing closed-loop algorithms. There are two general methods to implementing closed-loop algorithms to modulate neural activity: 1) Model-free algorithms that require a learning period to establish an optimal mapping between neural states and best therapeutic parameters, and 2) Model-based algorithms that use forward predictions of the neural system to determine the appropriate stimulation therapy to be administered. In this thesis, I will propose and test two closed-loop control schemes to control the brain activity to prevent epileptogenic activity while reducing stimulation energy. I will also present techniques to remove stimulation artifacts so that neural biomarkers can be measured while simultaneously applying stimulation. The methods I will present could potentially be implemented in next generation electrical brain stimulation hardware for seizure disorders and other neurological diseases

A Theory of Cheap Control in Embodied Systems

We present a framework for designing cheap control architectures for embodied agents. Our derivation is guided by the classical problem of universal approximation, whereby we explore the possibility of exploiting the agent's embodiment for a new and more efficient universal approximation of behaviors generated by sensorimotor control. This embodied universal approximation is compared with the classical non-embodied universal approximation. To exemplify our approach, we present a detailed quantitative case study for policy models defined in terms of conditional restricted Boltzmann machines. In contrast to non-embodied universal approximation, which requires an exponential number of parameters, in the embodied setting we are able to generate all possible behaviors with a drastically smaller model, thus obtaining cheap universal approximation. We test and corroborate the theory experimentally with a six-legged walking machine. The experiments show that the sufficient controller complexity predicted by our theory is tight, which means that the theory has direct practical implications. Keywords: cheap design, embodiment, sensorimotor loop, universal approximation, conditional restricted Boltzmann machineComment: 27 pages, 10 figure

The influence of dopamine on prediction, action and learning

In this thesis I explore functions of the neuromodulator dopamine in the context of autonomous learning and behaviour. I first investigate dopaminergic influence within a simulated agent-based model, demonstrating how modulation of synaptic plasticity can enable reward-mediated learning that is both adaptive and self-limiting. I describe how this mechanism is driven by the dynamics of agentenvironment interaction and consequently suggest roles for both complex spontaneous neuronal activity and specific neuroanatomy in the expression of early, exploratory behaviour. I then show how the observed response of dopamine neurons in the mammalian basal ganglia may also be modelled by similar processes involving dopaminergic neuromodulation and cortical spike-pattern representation within an architecture of counteracting excitatory and inhibitory neural pathways, reflecting gross mammalian neuroanatomy. Significantly, I demonstrate how combined modulation of synaptic plasticity and neuronal excitability enables specific (timely) spike-patterns to be recognised and selectively responded to by efferent neural populations, therefore providing a novel spike-timing based implementation of the hypothetical ‘serial-compound’ representation suggested by temporal difference learning. I subsequently discuss more recent work, focused upon modelling those complex spike-patterns observed in cortex. Here, I describe neural features likely to contribute to the expression of such activity and subsequently present novel simulation software allowing for interactive exploration of these factors, in a more comprehensive neural model that implements both dynamical synapses and dopaminergic neuromodulation. I conclude by describing how the work presented ultimately suggests an integrated theory of autonomous learning, in which direct coupling of agent and environment supports a predictive coding mechanism, bootstrapped in early development by a more fundamental process of trial-and-error learning

Perspectives on adaptive dynamical systems

Adaptivity is a dynamical feature that is omnipresent in nature, socio-economics, and technology. For example, adaptive couplings appear in various real-world systems, such as the power grid, social, and neural networks, and they form the backbone of closed-loop control strategies and machine learning algorithms. In this article, we provide an interdisciplinary perspective on adaptive systems. We reflect on the notion and terminology of adaptivity in different disciplines and discuss which role adaptivity plays for various fields. We highlight common open challenges and give perspectives on future research directions, looking to inspire interdisciplinary approaches