Model-based reinforcement learning: A survey

Reinforcement learning is an important branch of machine learning and artificial intelligence. Compared with traditional reinforcement learning, model-based reinforcement learning obtains the action of the next state by the model that has been learned, and then optimizes the policy, which greatly improves data efficiency. Based on the present status of research on model-based reinforcement learning at home and abroad, this paper comprehensively reviews the key techniques of model-based reinforcement learning, summarizes the characteristics, advantages and defects of each technology, and analyzes the application of model-based reinforcement learning in games, robotics and brain science

Adaptive Parameter Selection for Deep Brain Stimulation in Parkinson’s Disease

Each year, around 60,000 people are diagnosed with Parkinson’s disease (PD) and the economic burden of PD is at least $14.4 billion a year in the United States. Pharmaceutical costs for a Parkinson’s patient can be reduced from$12,000 to $6,000 per year with the addition of neuromodulation therapies such as Deep Brain Stimulation (DBS), transcranial Direct Current Stimulation (tDCS), Transcranial Magnetic Stimulation (TMS), etc. In neurodegenerative disorders such as PD, deep brain stimulation (DBS) is a desirable approach when the medication is less effective for treating the symptoms. DBS incorporates transferring electrical pulses to a specific tissue of the central nervous system and obtaining therapeutic results by modulating the neuronal activity of that region. The hyperkinetic symptoms of PD are associated with the ensembles of interacting oscillators that cause excess or abnormal synchronous behavior within the Basal Ganglia (BG) circuitry. Delayed feedback stimulation is a closed loop technique shown to suppress this synchronous oscillatory activity. Deep Brain Stimulation via delayed feedback is known to destabilize the complex intermittent synchronous states. Computational models of the BG network are often introduced to investigate the effect of delayed feedback high frequency stimulation on partially synchronized dynamics. In this work, we developed several computational models of four interacting nuclei of the BG as well as considering the Thalamo-Cortical local effects on the oscillatory dynamics. These models are able to capture the emergence of 34 Hz beta band oscillations seen in the Local Field Potential (LFP) recordings of the PD state. Traditional High Frequency Stimulations (HFS) has shown deficiencies such as strengthening the synchronization in case of highly fluctuating neuronal activities, increasing the energy consumed as well as the incapability of activating all neurons in a large-scale network. To overcome these drawbacks, we investigated the effects of the stimulation waveform and interphase delays on the overall efficiency and efficacy of DBS. We also propose a new feedback control variable based on the filtered and linearly delayed LFP recordings. The proposed control variable is then used to modulate the frequency of the stimulation signal rather than its amplitude. In strongly coupled networks, oscillations reoccur as soon as the amplitude of the stimulus signal declines. Therefore, we show that maintaining a fixed amplitude and modulating the frequency might ameliorate the desynchronization process, increase the battery lifespan and activate substantial regions of the administered DBS electrode. The charge balanced stimulus pulse itself is embedded with a delay period between its charges to grant robust desynchronization with lower amplitude needed. The efficiency and efficacy of the proposed Frequency Adjustment Stimulation (FAS) protocol in a delayed feedback method might contribute to further investigation of DBS modulations aspired to address a wide range of abnormal oscillatory behaviors observed in neurological disorders. Adaptive stimulation can open doors towards simultaneous stimulation with MRI recordings. We additionally propose a new pipeline to investigate the effect of Transcranial Magnetic Stimulation (TMS) on patient specific models. The pipeline allows us to generate a full head segmentation based on each individual MRI data. In the next step, the neurosurgeon can adaptively choose the proper location of stimulation and transmit accurate magnetic field with this pipeline

Neural Models of Normal and Abnormal Behavior: What Do Schizophrenia, Parkinsonism, Attention Deficit Disorder, and Depression Have in Common?

Defense Advanced Research Projects Agency and Office of Naval Research (N00014-95-1-0409); National Science Foundation (IRI-97-20333

Machine learning for large-scale wearable sensor data in Parkinson disease:concepts, promises, pitfalls, and futures

For the treatment and monitoring of Parkinson's disease (PD) to be scientific, a key requirement is that measurement of disease stages and severity is quantitative, reliable, and repeatable. The last 50 years in PD research have been dominated by qualitative, subjective ratings obtained by human interpretation of the presentation of disease signs and symptoms at clinical visits. More recently, “wearable,” sensor-based, quantitative, objective, and easy-to-use systems for quantifying PD signs for large numbers of participants over extended durations have been developed. This technology has the potential to significantly improve both clinical diagnosis and management in PD and the conduct of clinical studies. However, the large-scale, high-dimensional character of the data captured by these wearable sensors requires sophisticated signal processing and machine-learning algorithms to transform it into scientifically and clinically meaningful information. Such algorithms that “learn” from data have shown remarkable success in making accurate predictions for complex problems in which human skill has been required to date, but they are challenging to evaluate and apply without a basic understanding of the underlying logic on which they are based. This article contains a nontechnical tutorial review of relevant machine-learning algorithms, also describing their limitations and how these can be overcome. It discusses implications of this technology and a practical road map for realizing the full potential of this technology in PD research and practice

Open research issues on multi-models for complex technological systems

Abstract -We are going to report here about state of the art works on multi-models for complex technological systems both from the theoretical and practical point of view. A variety of algorithmic approaches (k-mean, dss, etc.) and applicative domains (wind farms, neurological diseases, etc.) are reported to illustrate the extension of the research area

Alzheimer’s And Parkinson’s Disease Classification Using Deep Learning Based On MRI: A Review

Neurodegenerative disorders present a current challenge for accurate diagnosis and for providing precise prognostic information. Alzheimer’s disease (AD) and Parkinson's disease (PD), may take several years to obtain a definitive diagnosis. Due to the increased aging population in developed countries, neurodegenerative diseases such as AD and PD have become more prevalent and thus new technologies and more accurate tests are needed to improve and accelerate the diagnostic procedure in the early stages of these diseases. Deep learning has shown significant promise in computer-assisted AD and PD diagnosis based on MRI with the widespread use of artificial intelligence in the medical domain. This article analyses and evaluates the effectiveness of existing Deep learning (DL)-based approaches to identify neurological illnesses using MRI data obtained using various modalities, including functional and structural MRI. Several current research issues are identified toward the conclusion, along with several potential future study directions

A Closed-Loop Brain Stimulation Control System Design Based on Brain-Machine Interface for Epilepsy

In this study, a closed-loop brain stimulation control system scheme for epilepsy seizure abatement is designed by brain-machine interface (BMI) technique. In the controller design process, the practical parametric uncertainties involving cerebral blood flow, glucose metabolism, blood oxygen level dependence, and electromagnetic disturbances in signal control are considered. An appropriate transformation is introduced to express the system in regular form for design and analysis. Then, sufficient conditions are developed such that the sliding motion is asymptotically stable. Combining Caputo fractional order definition and neural network (NN), a finite time fractional order sliding mode (FFOSM) controller is designed to guarantee reachability of the sliding mode. The stability and reachability analysis of the closed-loop tracking control system gives the guideline of parameter selection, and simulation results based on comprehensive comparisons are carried out to demonstrate the effectiveness of proposed approach

Doctor of Philosophy

dissertationIntegration of sensory inputs by the central nervous system (CNS) is necessary for adequate postural stability, but diminishes with age and is further impaired in Parkinson disease (PD). As a result, the CNS cannot appropriately weight sensory stimuli to facilitate postural responses to sudden changes in sensory input. Training the sensorimotor system to ignore or rapidly adapt to aberrant postural cues may improve postural control in PD. We evaluated the influence of acute and repeated exposure to galvanic vestibular stimulation (GVS) on postural responses during static and dynamic tasks to determine whether training improved these responses. We hypothesized that individuals with PD would demonstrate impaired postural recovery responses to acute GVS relative to healthy controls and that individuals with PD and healthy elders would demonstrate diminished adaptive responses to repeated GVS compared to young adults. Twelve individuals with PD (PD group), 15 healthy young adults (HY group), and 11 healthy elders (HE group) participated. Timing of GVS was randomly applied during each task. Fifteen acquisition and nine retention trials with GVS were compared to assess learning. The PD group took longer to stabilize their center of pressure (COP) in quiet stance following GVS acutely compared to controls. The PD and HE groups had lower sample entropy (SaEn) compared to the HY. Neither the PD nor HE groups demonstrated changes in SaEn or meaningful improvements in postural control during acquisition or retention. SaEn in the HY group acutely decreased and then increased at retention which coincided with a meaningful improvement in postural control. The PD group had impaired motor planning, postural preparation, and postural stability during a rise to toes task following acute GVS, but these constructs returned to baseline at later acquisition and retention time points. Controls suppressed GVS acutely Postural coordination decreased acutely in the PD group during tether release. This persisted and an adaptive trend in BOS transition was noted with repeated GVS exposure in this group. No changes were observed in the control groups. Taken together, these results demonstrated that acute GVS differentially affects postural control in individuals with PD. Our results support the hypothesis that reweighting of sensory stimuli is impaired in PD. We also show that individuals with PD are able to suppress attention to a vestibular illusion and demonstrate adaptive responses to a postural threat