Certainty equivalence and model uncertainty

Simon’s and Theil’s certainty equivalence property justifies a convenient algorithm for solving dynamic programming problems with quadratic objectives and linear transition laws: first, optimize under perfect foresight, then substitute optimal forecasts for unknown future values. A similar decomposition into separate optimization and forecasting steps prevails when a decision maker wants a decision rule that is robust to model misspecification. Concerns about model misspecification leave the first step of the algorithm intact and affect only the second step of forecasting the future. The decision maker attains robustness by making forecasts with a distorted model that twists probabilities relative to his approximating model. The appropriate twisting emerges from a two-player zero-sum dynamic game.

Comparing Strategies to Prevent Stroke and Ischemic Heart Disease in the Tunisian Population: Markov Modeling Approach Using a Comprehensive Sensitivity Analysis Algorithm.

Background. Mathematical models offer the potential to analyze and compare the effectiveness of very different interventions to prevent future cardiovascular disease. We developed a comprehensive Markov model to assess the impact of three interventions to reduce ischemic heart diseases (IHD) and stroke deaths: (i) improved medical treatments in acute phase, (ii) secondary prevention by increasing the uptake of statins, (iii) primary prevention using health promotion to reduce dietary salt consumption. Methods. We developed and validated a Markov model for the Tunisian population aged 35–94 years old over a 20-year time horizon. We compared the impact of specific treatments for stroke, lifestyle, and primary prevention on both IHD and stroke deaths. We then undertook extensive sensitivity analyses using both a probabilistic multivariate approach and simple linear regression (metamodeling). Results. The model forecast a dramatic mortality rise, with 111,134 IHD and stroke deaths (95% CI 106567 to 115048) predicted in 2025 in Tunisia. The salt reduction offered the potentially most powerful preventive intervention that might reduce IHD and stroke deaths by 27% (−30240 [−30580 to −29900]) compared with 1% for medical strategies and 3% for secondary prevention. The metamodeling highlighted that the initial development of a minor stroke substantially increased the subsequent probability of a fatal stroke or IHD death. Conclusions. The primary prevention of cardiovascular disease via a reduction in dietary salt consumption appeared much more effective than secondary or tertiary prevention approaches. Our simple but comprehensive model offers a potentially attractive methodological approach that might now be extended and replicated in other contexts and populations

A New Decision Support Framework for Managing Foot-and-mouth Disease Epidemics

Animal disease epidemics such as the foot-and-mouth disease (FMD) pose recurrent threat to countries with intensive livestock production. Efficient FMD control is crucial in limiting the damage of FMD epidemics and securing food production. Decision making in FMD control involves a hierarchy of decisions made at strategic, tactical, and operational levels. These decisions are interdependent and have to be made under uncertainty about future development of the epidemic. Addressing this decision problem, this paper presents a new decision-support framework based on multi-level hierarchic Markov processes (MLHMP). The MLHMP model simultaneously optimizes decisions at strategic, tactical, and operational levels, using Bayesian forecasting methods to model uncertainty and learning about the epidemic. As illustrated by the example, the framework is especially useful in contingency planning for future FMD epidemic

Approximate Models and Robust Decisions

Decisions based partly or solely on predictions from probabilistic models may be sensitive to model misspecification. Statisticians are taught from an early stage that "all models are wrong", but little formal guidance exists on how to assess the impact of model approximation on decision making, or how to proceed when optimal actions appear sensitive to model fidelity. This article presents an overview of recent developments across different disciplines to address this. We review diagnostic techniques, including graphical approaches and summary statistics, to help highlight decisions made through minimised expected loss that are sensitive to model misspecification. We then consider formal methods for decision making under model misspecification by quantifying stability of optimal actions to perturbations to the model within a neighbourhood of model space. This neighbourhood is defined in either one of two ways. Firstly, in a strong sense via an information (Kullback-Leibler) divergence around the approximating model. Or using a nonparametric model extension, again centred at the approximating model, in order to `average out' over possible misspecifications. This is presented in the context of recent work in the robust control, macroeconomics and financial mathematics literature. We adopt a Bayesian approach throughout although the methods are agnostic to this position

Planning with Information-Processing Constraints and Model Uncertainty in Markov Decision Processes

Information-theoretic principles for learning and acting have been proposed to solve particular classes of Markov Decision Problems. Mathematically, such approaches are governed by a variational free energy principle and allow solving MDP planning problems with information-processing constraints expressed in terms of a Kullback-Leibler divergence with respect to a reference distribution. Here we consider a generalization of such MDP planners by taking model uncertainty into account. As model uncertainty can also be formalized as an information-processing constraint, we can derive a unified solution from a single generalized variational principle. We provide a generalized value iteration scheme together with a convergence proof. As limit cases, this generalized scheme includes standard value iteration with a known model, Bayesian MDP planning, and robust planning. We demonstrate the benefits of this approach in a grid world simulation.Comment: 16 pages, 3 figure

Trading Safety Versus Performance: Rapid Deployment of Robotic Swarms with Robust Performance Constraints

In this paper we consider a stochastic deployment problem, where a robotic swarm is tasked with the objective of positioning at least one robot at each of a set of pre-assigned targets while meeting a temporal deadline. Travel times and failure rates are stochastic but related, inasmuch as failure rates increase with speed. To maximize chances of success while meeting the deadline, a control strategy has therefore to balance safety and performance. Our approach is to cast the problem within the theory of constrained Markov Decision Processes, whereby we seek to compute policies that maximize the probability of successful deployment while ensuring that the expected duration of the task is bounded by a given deadline. To account for uncertainties in the problem parameters, we consider a robust formulation and we propose efficient solution algorithms, which are of independent interest. Numerical experiments confirming our theoretical results are presented and discussed

Exploration vs Exploitation vs Safety: Risk-averse Multi-Armed Bandits

Motivated by applications in energy management, this paper presents the Multi-Armed Risk-Aware Bandit (MARAB) algorithm. With the goal of limiting the exploration of risky arms, MARAB takes as arm quality its conditional value at risk. When the user-supplied risk level goes to 0, the arm quality tends toward the essential infimum of the arm distribution density, and MARAB tends toward the MIN multi-armed bandit algorithm, aimed at the arm with maximal minimal value. As a first contribution, this paper presents a theoretical analysis of the MIN algorithm under mild assumptions, establishing its robustness comparatively to UCB. The analysis is supported by extensive experimental validation of MIN and MARAB compared to UCB and state-of-art risk-aware MAB algorithms on artificial and real-world problems.Comment: 16 page