Improving Reinforcement Learning Techniques for Medical Decision Making

Reinforcement learning (RL) is a powerful tool for developing personalized treatment regimens from healthcare data. In RL, an agent samples experiences from an environment (such as a model of patient health) to learn a policy that maximizes long-term reward. This dissertation proposes methodological and practical developments in the application of RL to treatment planning problems. First, we develop a novel time series model for simulating patient health states from observed clinical data. We use a generative neural network architecture that learns a direct mapping between distributions over clinical measurements at adjacent time points. We show that this model produces realistic patient trajectories and can be paired with on-policy RL to learn effective treatment policies. Second, we develop a novel extension of hidden Markov models, which are commonly used to model and predict patient health states. Specifically, we develop a special case of recurrent neural networks with the same likelihood function as a corresponding discrete-observation hidden Markov model. We demonstrate how combining our model with other predictive neural networks improves disease forecasting and offers novel clinical interpretations compared with a standard hidden Markov model. Third, we develop a method for selecting high-performing reinforcement learning-based treatment policies for underrepresented patient subpopulations using limited observations. Our method learns a probability distribution over treatment policies from a reference patient group, then adapts its recommendations using limited data from an underrepresented patient group. We show that our method outperforms state-of-the-art benchmarks in selecting effective treatment policies for patients with non-typical clinical characteristics, and predicting these patients\u27 outcomes under its policies. Finally, we use RL to optimize medication regimens for Parkinson\u27s disease patients using high-frequency wearable sensor data. We build an environment model of how patients\u27 symptoms respond to medication, then use RL to recommend optimal medication types, timing, and dosages for each patient. We show that these patient-specific RL-prescribed medication regimens outperform physician-prescribed regimens and provide clinically defensible treatment strategies. Our framework also enables physicians to identify patients who could could switch to lower-frequency regimens for improved adherence, and to identify patients who may be candidates for advanced therapies

A survey of models for inference of gene regulatory networks

In this article, I present the biological backgrounds of microarray, ChIP-chip and ChIPSeq technologies and the application of computational methods in reverse engineering of gene regulatory networks (GRNs). The most commonly used GRNs models based on Boolean networks, Bayesian networks, relevance networks, differential and difference equations are described. A novel model for integration of prior biological knowledge in the GRNs inference is presented, too. The advantages and disadvantages of the described models are compared. The GRNs validation criteria are depicted. Current trends and further directions for GRNs inference using prior knowledge are given at the end of the paper

Investigating the Causal Mechanisms of Symptom Recovery in Chronic Whiplash-associated Disorders Using Bayesian Networks

Objectives: The present study’s objective was to understand the causal mechanisms underpinning the recovery of individuals with whiplash-associated disorders (WAD). We applied Bayesian Networks (BN) to answer 2 study aims: (1) to identify the causal mechanism(s) of recovery underpinning neck-specific exercise (NSE), and (2) quantify if the cyclical pathway of the fear-avoidance model (FAM) is supported by the present data. Materials and Methods: We analyzed a prospective cohort data set of 216 individuals with chronic WAD. Fifteen variables were used to build a BN model: treatment group (NSE with or without a behavioral approach, or general physical activity), muscle endurance, range of motion, hand strength, neck proprioception, pain catastrophizing, fear, anxiety, depression, self-efficacy, perceived work ability, disability, pain intensity, sex, and follow-up time. Results: The BN model showed that neck pain reduction rate was greater after NSE compared with physical activity prescription (β=0.59 points per month [P<0.001]) only in the presence of 2 mediators: global neck muscle endurance and perceived work ability. We also found the following pathway of variables that constituted the FAM: anxiety, followed by depressive symptoms, fear, catastrophizing, self-efficacy, and consequently pain. Conclusions: e uncovered 2 mediators that explained the mechanisms of effect behind NSE, and proposed an alternative FAM pathway. The present study is the first to apply BN modelling to understand the causal mechanisms of recovery in WAD. In doing so, it is anticipated that such analytical methods could increase the precision of treatment of individuals with chronic WAD

Sparse Representation-Based Framework for Preprocessing Brain MRI

This thesis addresses the use of sparse representations, specifically Dictionary Learning and Sparse Coding, for pre-processing brain MRI, so that the processed image retains the fine details of the original image, to improve the segmentation of brain structures, to assess whether there is any relationship between alterations in brain structures and the behavior of young offenders. Denoising an MRI while keeping fine details is a difficult task; however, the proposed method, based on sparse representations, NLM, and SVD can filter noise while prevents blurring, artifacts, and residual noise. Segmenting an MRI is a non-trivial task; because normally the limits between regions in these images may be neither clear nor well defined, due to the problems which affect MRI. However, this method, from both the label matrix of the segmented MRI and the original image, yields a new improved label matrix in which improves the limits among regions.DoctoradoDoctor en Ingeniería de Sistemas y Computació

Utilizing Knowledge Bases In Information Retrieval For Clinical Decision Support And Precision Medicine

Accurately answering queries that describe a clinical case and aim at finding articles in a collection of medical literature requires utilizing knowledge bases in capturing many explicit and latent aspects of such queries. Proper representation of these aspects needs knowledge-based query understanding methods that identify the most important query concepts as well as knowledge-based query reformulation methods that add new concepts to a query. In the tasks of Clinical Decision Support (CDS) and Precision Medicine (PM), the query and collection documents may have a complex structure with different components, such as disease and genetic variants that should be transformed to enable an effective information retrieval. In this work, we propose methods for representing domain-specific queries based on weighted concepts of different types whether exist in the query itself or extracted from the knowledge bases and top retrieved documents. Besides, we propose an optimization framework, which allows unifying query analysis and expansion by jointly determining the importance weights for the query and expansion concepts depending on their type and source. We also propose a probabilistic model to reformulate the query given genetic information in the query and collection documents. We observe significant improvement of retrieval accuracy will be obtained for our proposed methods over state-of-the-art baselines for the tasks of clinical decision support and precision medicine

Biomedical applications of belief networks

Biomedicine is an area in which computers have long been expected to play a significant role. Although many of the early claims have proved unrealistic, computers are gradually becoming accepted in the biomedical, clinical and research environment. Within these application areas, expert systems appear to have met with the most resistance, especially when applied to image interpretation.In order to improve the acceptance of computerised decision support systems it is necessary to provide the information needed to make rational judgements concerning the inferences the system has made. This entails an explanation of what inferences were made, how the inferences were made and how the results of the inference are to be interpreted. Furthermore there must be a consistent approach to the combining of information from low level computational processes through to high level expert analyses.nformation from low level computational processes through to high level expert analyses. Until recently ad hoc formalisms were seen as the only tractable approach to reasoning under uncertainty. A review of some of these formalisms suggests that they are less than ideal for the purposes of decision making. Belief networks provide a tractable way of utilising probability theory as an inference formalism by combining the theoretical consistency of probability for inference and decision making, with the ability to use the knowledge of domain experts.nowledge of domain experts. The potential of belief networks in biomedical applications has already been recog¬ nised and there has been substantial research into the use of belief networks for medical diagnosis and methods for handling large, interconnected networks. In this thesis the use of belief networks is extended to include detailed image model matching to show how, in principle, feature measurement can be undertaken in a fully probabilistic way. The belief networks employed are usually cyclic and have strong influences between adjacent nodes, so new techniques for probabilistic updating based on a model of the matching process have been developed.An object-orientated inference shell called FLAPNet has been implemented and used to apply the belief network formalism to two application domains. The first application is model-based matching in fetal ultrasound images. The imaging modality and biological variation in the subject make model matching a highly uncertain process. A dynamic, deformable model, similar to active contour models, is used. A belief network combines constraints derived from local evidence in the image, with global constraints derived from trained models, to control the iterative refinement of an initial model cue.In the second application a belief network is used for the incremental aggregation of evidence occurring during the classification of objects on a cervical smear slide as part of an automated pre-screening system. A belief network provides both an explicit domain model and a mechanism for the incremental aggregation of evidence, two attributes important in pre-screening systems.Overall it is argued that belief networks combine the necessary quantitative features required of a decision support system with desirable qualitative features that will lead to improved acceptability of expert systems in the biomedical domain