Reinforcement learning application in diabetes blood glucose control: A systematic review

Background: Reinforcement learning (RL) is a computational approach to understanding and automating goal-directed learning and decision-making. It is designed for problems which include a learning agent interacting with its environment to achieve a goal. For example, blood glucose (BG) control in diabetes mellitus (DM), where the learning agent and its environment are the controller and the body of the patient respectively. RL algorithms could be used to design a fully closed-loop controller, providing a truly personalized insulin dosage regimen based exclusively on the patient’s own data. Objective: In this review we aim to evaluate state-of-the-art RL approaches to designing BG control algorithms in DM patients, reporting successfully implemented RL algorithms in closed-loop, insulin infusion, decision support and personalized feedback in the context of DM. Methods: An exhaustive literature search was performed using different online databases, analyzing the literature from 1990 to 2019. In a first stage, a set of selection criteria were established in order to select the most relevant papers according to the title, keywords and abstract. Research questions were established and answered in a second stage, using the information extracted from the articles selected during the preliminary selection. Results: The initial search using title, keywords, and abstracts resulted in a total of 404 articles. After removal of duplicates from the record, 347 articles remained. An independent analysis and screening of the records against our inclusion and exclusion criteria defined in Methods section resulted in removal of 296 articles, leaving 51 relevant articles. A full-text assessment was conducted on the remaining relevant articles, which resulted in 29 relevant articles that were critically analyzed. The inter-rater agreement was measured using Cohen Kappa test, and disagreements were resolved through discussion. Conclusions: The advances in health technologies and mobile devices have facilitated the implementation of RL algorithms for optimal glycemic regulation in diabetes. However, there exists few articles in the literature focused on the application of these algorithms to the BG regulation problem. Moreover, such algorithms are designed for control tasks as BG adjustment and their use have increased recently in the diabetes research area, therefore we foresee RL algorithms will be used more frequently for BG control in the coming years. Furthermore, in the literature there is a lack of focus on aspects that influence BG level such as meal intakes and physical activity (PA), which should be included in the control problem. Finally, there exists a need to perform clinical validation of the algorithms

In-Silico Evaluation of Glucose Regulation Using Policy Gradient Reinforcement Learning for Patients with Type 1 Diabetes Mellitus

In this paper, we test and evaluate policy gradient reinforcement learning for automated blood glucose control in patients with Type 1 Diabetes Mellitus. Recent research has shown that reinforcement learning is a promising approach to accommodate the need for individualized blood glucose level control algorithms. The motivation for using policy gradient algorithms comes from the fact that adaptively administering insulin is an inherently continuous task. Policy gradient algorithms are known to be superior in continuous high-dimensional control tasks. Previously, most of the approaches for automated blood glucose control using reinforcement learning has used a finite set of actions. We use the Trust-Region Policy Optimization algorithm in this work. It represents the state of the art for deep policy gradient algorithms. The experiments are carried out in-silico using the Hovorka model, and stochastic behavior is modeled through simulated carbohydrate counting errors to illustrate the full potential of the framework. Furthermore, we use a model-free approach where no prior information about the patient is given to the algorithm. Our experiments show that the reinforcement learning agent is able to compete with and sometimes outperform state-of-the-art model predictive control in blood glucose regulation

On-line policy learning and adaptation for real-time personalization of an artificial pancreas

The dynamic complexity of the glucose-insulin metabolism in diabetic patients is the main obstacle towards widespread use of an artificial pancreas. The significant level of subject-specific glycemic variability requires continuously adapting the control policy to successfully face daily changes in patient´s metabolism and lifestyle. In this paper, an on-line selective reinforcement learning algorithm that enables real-time adaptation of a control policy based on ongoing interactions with the patient so as to tailor the artificial pancreas is proposed. Adaptation includes two online procedures: on-line sparsification and parameter updating of the Gaussian process used to approximate the control policy. With the proposed sparsification method, the support data dictionary for on-line learning is modified by checking if in the arriving data stream there exists novel information to be added to the dictionary in order to personalize the policy. Results obtained in silico experiments demonstrate that on-line policy learning is both safe and efficient for maintaining blood glucose variability within the normoglycemic range.Fil: de Paula, Mariano. Universidad Nacional del Centro de la Provincia de Buenos Aires. Facultad de Ingeniería Olavarria. Departamento de Electromecánica. Grupo INTELYMEC; Argentina. Universidad Nacional del Centro de la Pcia.de Bs.as.. Centro de Investigaciones En Fisica E Ingenieria del Centro de la Provincia de Buenos Aires. - Consejo Nacional de Investigaciones Cientificas y Tecnicas. Centro Cientifico Tecnologico Conicet - Tandil. Centro de Investigaciones En Fisica E Ingenieria del Centro de la Provincia de Buenos Aires. - Provincia de Buenos Aires. Gobernacion. Comision de Invest.cientificas. Centro de Investigaciones En Fisica E Ingenieria del Centro de la Provincia de Buenos Aires; ArgentinaFil: Acosta, Gerardo Gabriel. Universidad Nacional del Centro de la Provincia de Buenos Aires. Facultad de Ingenieria Olavarria; Argentina. Consejo Nacional de Investigaciones Científicas y Técnicas; ArgentinaFil: Martinez, Ernesto Carlos. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Santa Fe. Instituto de Desarrollo y Diseño. Universidad Tecnológica Nacional. Facultad Regional Santa Fe. Instituto de Desarrollo y Diseño; Argentin

The Cost-Effectiveness of Improving Diabetes Care in U.S. Federally Qualified Community Health Centers

Objective. To estimate the incremental cost-effectiveness of improving diabetes care with the Health Disparities Collaborative (HDC), a national collaborative quality improvement (QI) program conducted in community health centers (HCs). Data Sources/Study Settings. Data regarding the impact of the Diabetes HDC program came from a serial cross-sectional follow-up study (1998, 2000, 2002) of the program in 17 Midwestern HCs. Data inputs for the simulation model of diabetes came from the latest clinical trials and epidemiological studies. Study Design. We conducted a societal cost-effectiveness analysis, incorporating data from QI program evaluation into a Monte Carlo simulation model of diabetes. Data Collections/Extraction Methods. Data on diabetes care processes and risk factor levels were extracted from medical charts of randomly selected patients. Principal Findings. From 1998 to 2002, multiple processes of care (e.g., glycosylated hemoglobin testing [HbA1C] [71 -\u3e 92 percent] and ACE inhibitor prescribing [33 -\u3e 55 percent]) and risk factor levels (e.g., 1998 mean HbA1C 8.53 percent, mean difference 0.45 percent [95 percent confidence intervals -0.72, -0.17]) improved significantly. With these improvements, the HDC was estimated to reduce the lifetime incidence of blindness (17 -\u3e 15 percent), end-stage renal disease (18 -\u3e 15 percent), and coronary artery disease (28 -\u3e 24 percent). The average improvement in quality-adjusted life year (QALY) was 0.35 and the incremental cost-effectiveness ratio was $33,386/QALY. Conclusions. During the first 4 years of the HDC, multiple improvements in diabetes care were observed. If these improvements are maintained or enhanced over the lifetime of patients, the HDC program will be cost-effective for society based on traditionally accepted thresholds

Precision Medicine: Viable Pathways to Address Existing Research Gaps

Precision Medicine (PM) seeks to customize medical treatments for patients based on measurable and identifiable characteristics. Unlike personalized medicine, this effort is not intended to result in tailored care for each patient. Instead, this effort seeks to improve overall care within the medical domain by shifting the focus from one-size-fits-all care to optimized care for specified subgroups. In order for the benefits of PM to be expeditiously realized, the diverse skills sets of the scientific community must be brought to bear on the problem. This research effort explores the intersection of quality engineering (QE) and healthcare to outline how existing methodologies within the QE field could support existing PM research goals. Specifically this work examines how to determine the value of patient characteristics for use in disease prediction models with select machine learning algorithms, proposes a method to incorporate patient risk into treatment decisions through the development of performance functions, and investigates the potential impact of incorrect assumptions on estimation methods used in optimization models

Real-time monitoring and forecasting of time series in healthcare applications

Phd ThesisType II diabetes is an increasingly common disease, but one in which the effects suffered by patients, such as hyperglycaemia, can be improved through careful monitoring and control of the factors that influence blood glucose levels. Advances in the Internet of Things (IoT) have made monitoring a person’s glucose levels more accessible, in that a continuous glucose monitoring (CGM) device in the form of a small sensor can be used to regularly report glucose levels to a bluetooth device, without the need for human intervention. Modelling the data from CGM devices online allows for short-term forecasts to be made that can assist in making real-time decisions regarding interventions to improve future glucose levels, such as behavioural changes. Additional data to monitor how active a person is can easily be collected by wrist-worn accelerometer devices. As activity levels directly impact glucose levels, bivariate models between glucose and activity data aim to provide improved forecasts. State space models are fitted to glucose data and activity data using a Bayesian modelling framework. The posterior distributions of model parameters are learned via Markov chain Monte Carlo (MCMC) methods. High frequency (100 Hz), tri-axial accelerometer data are reported alongside glucose observations recorded at five minute intervals and are transformed into univariate activity summaries. Discrete-valued state space models, known as hidden Markov models (HMMs), are used to classify the observations from the different activity summaries into activity intensities. Normal and skew Normal withinstate distributions are explored to better fit the observed activity summaries, as well as fitting models to transformations of the summaries where possible to reduce the skewness in the data. Gaussian state space models, known as dynamic linear models (DLMs), are explored to describe glucose levels, incorporating seasonal and autoregressive (AR) components. The results from these models then provide the basis for bivariate models that incorporate known activity states. This additional information is included in the DLMs as a regression covariate, which is formed by a weighted sum of lagged activity zones. Models between glucose levels and lagged carbohydrate intake are also considered, to better understand the effects of activity and food on glucose levels. A second application area is considered as an example of improved predictive performance where an influential variable is known alongside the quantity of interest. The production levels of liquid natural gas (LNG) at a gas plant are modelled by a DLM, with a regression on atmospheric temperature. The models are fitted in a frequentist framework for simplicit

Challenges in biomedical data science: data-driven solutions to clinical questions

Data are influencing every aspect of our lives, from our work activities, to our spare time and even to our health. In this regard, medical diagnosis and treatments are often supported by quantitative measures and observations, such as laboratory tests, medical imaging or genetic analysis. In medicine, as well as in several other scientific domains, the amount of data involved in each decision-making process has become overwhelming. The complexity of the phenomena under investigation and the scale of modern data collections has long superseded human analysis and insights potential

Machine learning and computational methods to identify molecular and clinical markers for complex diseases – case studies in cancer and obesity

In biomedical research, applied machine learning and bioinformatics are the essential disciplines heavily involved in translating data-driven findings into medical practice. This task is especially accomplished by developing computational tools and algorithms assisting in detection and clarification of underlying causes of the diseases. The continuous advancements in high-throughput technologies coupled with the recently promoted data sharing policies have contributed to presence of a massive wealth of data with remarkable potential to improve human health care. In concordance with this massive boost in data production, innovative data analysis tools and methods are required to meet the growing demand. The data analyzed by bioinformaticians and computational biology experts can be broadly divided into molecular and conventional clinical data categories. The aim of this thesis was to develop novel statistical and machine learning tools and to incorporate the existing state-of-the-art methods to analyze bio-clinical data with medical applications. The findings of the studies demonstrate the impact of computational approaches in clinical decision making by improving patients risk stratification and prediction of disease outcomes. This thesis is comprised of five studies explaining method development for 1) genomic data, 2) conventional clinical data and 3) integration of genomic and clinical data. With genomic data, the main focus is detection of differentially expressed genes as the most common task in transcriptome profiling projects. In addition to reviewing available differential expression tools, a data-adaptive statistical method called Reproducibility Optimized Test Statistic (ROTS) is proposed for detecting differential expression in RNA-sequencing studies. In order to prove the efficacy of ROTS in real biomedical applications, the method is used to identify prognostic markers in clear cell renal cell carcinoma (ccRCC). In addition to previously known markers, novel genes with potential prognostic and therapeutic role in ccRCC are detected. For conventional clinical data, ensemble based predictive models are developed to provide clinical decision support in treatment of patients with metastatic castration resistant prostate cancer (mCRPC). The proposed predictive models cover treatment and survival stratification tasks for both trial-based and realworld patient cohorts. Finally, genomic and conventional clinical data are integrated to demonstrate the importance of inclusion of genomic data in predictive ability of clinical models. Again, utilizing ensemble-based learners, a novel model is proposed to predict adulthood obesity using both genetic and social-environmental factors. Overall, the ultimate objective of this work is to demonstrate the importance of clinical bioinformatics and machine learning for bio-clinical marker discovery in complex disease with high heterogeneity. In case of cancer, the interpretability of clinical models strongly depends on predictive markers with high reproducibility supported by validation data. The discovery of these markers would increase chance of early detection and improve prognosis assessment and treatment choice