Reinforcement Learning for Low Probability High Impact Risks

We demonstrate a method of reinforcement learning that uses training in simulation. Our system generates an estimate of the potential reward and danger of each action as well as a measure of the uncertainty present in both. The system generates this by seeking out not only rewarding actions but also dangerous ones in the simulated training. During runtime our system is able to use this knowledge to avoid risks while accomplishing its tasks

Statistical Inference Utilizing Agent Based Models

<p>Agent-based models (ABMs) are computational models used to simulate the behaviors, </p><p>actionsand interactions of agents within a system. The individual agents </p><p>each have their own set of assigned attributes and rules, which determine</p><p>their behavior within the ABM system. These rules can be</p><p>deterministic or probabilistic, allowing for a great deal of</p><p>flexibility. ABMs allow us to</p><p>observe how the behaviors of the individual agents affect the system</p><p>as a whole and if any emergent structure develops within the</p><p>system. Examining rule sets in conjunction with corresponding emergent</p><p>structure shows how small-scale changes can</p><p>affect large-scale outcomes within the system. Thus, we can better</p><p>understand and predict the development and evolution of systems of</p><p>interest. </p><p>ABMs have become ubiquitous---they used in business</p><p>(virtual auctions to select electronic ads for display), atomospheric</p><p>science (weather forecasting), and public health (to model epidemics).</p><p>But there is limited understanding of the statistical properties of</p><p>ABMs. Specifically, there are no formal procedures</p><p>for calculating confidence intervals on predictions, nor for</p><p>assessing goodness-of-fit, nor for testing whether a specific</p><p>parameter (rule) is needed in an ABM.</p><p>Motivated by important challenges of this sort, </p><p>this dissertation focuses on developing methodology for uncertainty</p><p>quantification and statistical inference in a likelihood-free context</p><p>for ABMs. </p><p>Chapter 2 of the thesis develops theory related to ABMs, </p><p>including procedures for model validation, assessing model </p><p>equivalence and measuring model complexity. </p><p>Chapters 3 and 4 of the thesis focuses on two approaches </p><p>for performing likelihood-free inference involving ABMs, </p><p>which is necessary because of the intractability of the </p><p>likelihood function due to the variety of input rules and </p><p>the complexity of outputs.</p><p>Chapter 3 explores the use of </p><p>Gaussian Process emulators in conjunction with ABMs to perform </p><p>statistical inference. This draws upon a wealth of research on emulators, </p><p>which find smooth functions on lower-dimensional Euclidean spaces that approximate</p><p>the ABM. Emulator methods combine observed data with output from ABM</p><p>simulations, using these</p><p>to fit and calibrate Gaussian-process approximations. </p><p>Chapter 4 discusses Approximate Bayesian Computation for ABM inference, </p><p>the goal of which is to obtain approximation of the posterior distribution </p><p>of some set of parameters given some observed data. </p><p>The final chapters of the thesis demonstrates the approaches </p><p>for inference in two applications. Chapter 5 presents application models the spread </p><p>of HIV based on detailed data on a social network of men who have sex with</p><p>men (MSM) in southern India. Use of an ABM</p><p>will allow us to determine which social/economic/policy </p><p>factors contribute to thetransmission of the disease. </p><p>We aim to estimate the effect that proposed medical interventions will</p><p>have on the spread of HIV in this community. </p><p>Chapter 6 examines the function of a heroin market </p><p>in the Denver, Colorado metropolitan area. Extending an ABM </p><p>developed from ethnographic research, we explore a procedure </p><p>for reducing the model, as well as estimating posterior </p><p>distributions of important quantities based on simulations.</p>Dissertatio

ABC Reinforcement Learning

We introduce a simple, general framework for likelihood-free Bayesian reinforcement learning, through Approximate Bayesian Computation (ABC). The advantage is that we only require a prior distribution on a class of simulators. This is useful when a probabilistic model of the underlying process is too complex to formulate, but where detailed simulation models are available. ABC-RL allows the use of any Bayesian reinforcement learning technique in this case. It can be seen as an extension of simulation methods to both planning and inference. We experimentally demonstrate the potential of this approach in a comparison with LSPI. Finally, we introduce a theorem showing that ABC is sound. 1