Markov Decision Processes

The theory of Markov Decision Processes is the theory of controlled Markov chains. Its origins can be traced back to R. Bellman and L. Shapley in the 1950\u27s. During the decades of the last century this theory has grown dramatically. It has found applications in various areas like e.g. computer science, engineering, operations research, biology and economics. In this article we give a short introduction to parts of this theory. We treat Markov Decision Processes with finite and infinite time horizon where we will restrict the presentation to the so-called (generalized) negative case. Solution algorithms like Howard\u27s policy improvement and linear programming are also explained. Various examples show the application of the theory. We treat stochastic linear-quadratic control problems, bandit problems and dividend pay-out problems

Optimal control of single-server fluid networks

We consider a stochastic single server fluid network with both a discounted reward and a cost structure. It can be shown that the optimal policy is a priority index policy. The indices coincide with the optimal indices in a Semi-Markovian Klimov problem. Several special cases like single server re-entrant fluid lines are considered. The approach we use is based on sample path arguments and Pontryagins maximum principle

Optimal control of piecewise deterministic Markov processes with finite time horizon

In this paper we study controlled Piecewise Deterministic Markov Processes with finite time horizon and unbounded rewards. Using an embedding procedure we reduce these problems to discrete-time Markov Decision Processes. Under some continuity and compactness conditions we establish the existence of an optimal policy and show that the value function is the unique solution of the Bellman equation. It is remarkable that this statement is true for unbounded rewards and without any contraction assumptions. Further conditions imply the existence of optimal nonrelaxed controls. We highlight our findings by two examples from financial mathematics

Control improvement for jump-diffusion processes with applications to finance

We consider stochastic control problems with jump-diffusion processes and formulate an algorithm which produces, starting from a given admissible control Pi, a new control with a better value. If no improvement is possible, then Pi is optimal. Such an algorithm is well-known for discrete-time Markov Decision Problems under the name Howard’s policy improvement algorithm. The idea can be traced back to Bellman. Here we show with the help of martingale techniques that such an algorithm can also be formulated for stochastic control problems with jump-diffusion processes. As an application we derive some interesting results in portfolio optimization

MDP Algorithms for portfolio optimization problems in pure jump markets

We consider the problem of maximizing the expected utility of the terminal wealth of a portfolio in a continuous-time pure jump market with general utility function. This leads to an optimal control problem for Piecewise Deterministic Markov Processes. Using an embedding procedure we solve the problem by looking at a discrete-time contracting Markov Decision Process. Our aim is to show that this point of view has a number of advantages, in particular as far as computational aspects are concerned. We characterize the value function as the unique fixed point of the dynamic programming operator and prove the existence of optimal portfolios. Moreover, we show that value iteration as well as Howard\u27s policy improvement algorithm work. Finally we give error bounds when the utility function is approximated and when we discretize the state space. A numerical example is presented and our approach is compared to the approximating Markov chain method

IS THE END-TIDAL PARTIAL PRESSURE OF ISOFLURANE A GOOD PREDICTOR OF ITS ARTERIAL PARTIAL PRESSURE?

End-tidal partial pressure of isoflurane (PE′iso) may be used as a measure of anaesthetic depth. During uptake, an arterial partial pressure (Paiso) which is considerably less than PE′iso(Paiso/PE′iso<<1) leads to underestimation of depth of anaesthesia and, during elimination, PE′iso/Paiso<<1 will lead to an overestimation of anaesthetic depth. We measured Paiso/PE′iso during a 60-min uptake period of 1% isoflurane and PE′iso/Paiso during the subsequent 60-min elimination period in 26 patients (age 13-88 yr, ASA I-III) undergoing various surgical procedures. After 15 min of isoflurane uptake, Paiso/PE′iso of 26 patients was mean 0.78 (SD 0.10) and this increased only marginally at 60 min (0.79 (0.09)), whereas during elimination, PE′iso/Paiso was in the range 0.79 (0.14)-0.83 (0.11). Predictability of Paiso in a given patient is hindered by the high SD of Paiso/PE′iso and PE′iso/Paiso, but it may be improved by taking into account age, ASA physical status category, vital capacity, inspired minus end-tidal isoflurane partial pressure and arterial minus end-tidal carbon dioxide partial pressure during uptake; and obesity, end-tidal isoflurane partial pressure and arterial minus end-tidal carbon dioxide partial pressure during elimination. However, even with multiple regression analysis (to account for the various possible variables), clinically useful prediction of Paiso/PE′iso and PE′iso/Paiso in a particular patient is not possible (residual SD 0.084 and 0.113, respectively

ASA Status, NPPA/NPPB Haplotype and Coronary Artery Disease Have an Impact on BNP/NT-proBNP Plasma Levels.

Plasma concentrations of natriuretic peptides (NP) contribute to risk stratification and management of patients undergoing non-cardiac surgery. However, genetically determined variability in the levels of these biomarkers has been described previously. In the perioperative setting, genetic contribution to NP plasma level variability has not yet been determined. A cohort of 427 patients presenting for non-cardiac surgery was genotyped for single-nucleotide polymorphisms (SNPs) from the NPPA/NPPB locus. Haplotype population frequencies were estimated and adjusted haplotype trait associations for brain natriuretic peptide (BNP) and amino-terminal pro natriuretic peptide (NT-proBNP) were calculated. Five SNPs were included in the analysis. Compared to the reference haplotype TATAT (rs198358, rs5068, rs632793, rs198389, rs6676300), haplotype CACGC, with an estimated frequency of 4%, showed elevated BNP and NT-proBNP plasma concentrations by 44% and 94%, respectively. Haplotype CGCGC, with an estimated frequency of 9%, lowered NT-proBNP concentrations by 28%. ASA classification status III and IV, as well as coronary artery disease, were the strongest predictors of increased NP plasma levels. Inclusion of genetic information might improve perioperative risk stratification of patients based on adjusted thresholds of NP plasma levels

Monitoring of immune activation using biochemical changes in a porcine model of cardiac arrest.

In animal models, immune activation is often difficult to assess because of the limited availability of specific assays to detect cytokine activities. In human monocytes/macrophages, interferon-gamma induces increased production of neopterin and an enhanced activity of indoleamine 2,3-dioxygenase, which degrades tryptophan via the kynurenine pathway. Therefore, monitoring of neopterin concentrations and of tryptophan degradation can serve to detect the extent of T helper cell 1-type immune activation during cellular immune response in humans. In a porcine model of cardiac arrest, we examined the potential use of neopterin measurements and determination of the tryptophan degradation rate as a means of estimating the extent of immune activation. Urinary neopterin concentrations were measured with high-performance liquid chromatography (HPLC) and radioimmunoassay (RIA) (BRAHMS Diagnostica, Berlin, Germany). Serum and plasma tryptophan and kynurenine concentrations were also determined using HPLC. Serum and urine neopterin concentrations were not detectable with HPLC in these specimens, whereas RIA gave weakly (presumably false) positive results. The mean serum tryptophan concentration was 39.0 +/- 6.2 micromol/l, and the mean kynurenine concentration was 0.85 +/- 0.33 micromol/l. The average kynurenine-per-tryptophan quotient in serum was 21.7 +/- 8.4 nmol/micromol, and that in plasma was 20.7 +/- 9.5 nmol/micromol (n = 7), which corresponds well to normal values in humans. This study provides preliminary data to support the monitoring of tryptophan degradation but not neopterin concentrations as a potential means of detecting immune activation in a porcine model. The kynurenine-per-tryptophan quotient may serve as a short-term measurement of immune activation and hence permit an estimate of the extent of immune activation