Improving Reinforcement Learning Techniques for Medical Decision Making

Reinforcement learning (RL) is a powerful tool for developing personalized treatment regimens from healthcare data. In RL, an agent samples experiences from an environment (such as a model of patient health) to learn a policy that maximizes long-term reward. This dissertation proposes methodological and practical developments in the application of RL to treatment planning problems. First, we develop a novel time series model for simulating patient health states from observed clinical data. We use a generative neural network architecture that learns a direct mapping between distributions over clinical measurements at adjacent time points. We show that this model produces realistic patient trajectories and can be paired with on-policy RL to learn effective treatment policies. Second, we develop a novel extension of hidden Markov models, which are commonly used to model and predict patient health states. Specifically, we develop a special case of recurrent neural networks with the same likelihood function as a corresponding discrete-observation hidden Markov model. We demonstrate how combining our model with other predictive neural networks improves disease forecasting and offers novel clinical interpretations compared with a standard hidden Markov model. Third, we develop a method for selecting high-performing reinforcement learning-based treatment policies for underrepresented patient subpopulations using limited observations. Our method learns a probability distribution over treatment policies from a reference patient group, then adapts its recommendations using limited data from an underrepresented patient group. We show that our method outperforms state-of-the-art benchmarks in selecting effective treatment policies for patients with non-typical clinical characteristics, and predicting these patients\u27 outcomes under its policies. Finally, we use RL to optimize medication regimens for Parkinson\u27s disease patients using high-frequency wearable sensor data. We build an environment model of how patients\u27 symptoms respond to medication, then use RL to recommend optimal medication types, timing, and dosages for each patient. We show that these patient-specific RL-prescribed medication regimens outperform physician-prescribed regimens and provide clinically defensible treatment strategies. Our framework also enables physicians to identify patients who could could switch to lower-frequency regimens for improved adherence, and to identify patients who may be candidates for advanced therapies

Personalization of health interventions using cluster-based reinforcement learning

Research has shown that personalization of health interventions can contribute to an improved effectiveness. Reinforcement learning algorithms can be used to perform such tailoring. In this paper, we present a cluster-based reinforcement learning approach which learns optimal policies for groups of users. Such an approach can speed up the learning process while still giving a level of personalization. We apply both online and batch learning to learn policies over the clusters and introduce a publicly available simulator which we have developed to evaluate the approach. The results show batch learning significantly outperforms online learning. Furthermore, near-optimal clustering is found which proves to be beneficial in learning significantly better policies compared to learning per user and learning across all users