Markov Decision Processes with Average-Value-at-Risk criteria

We investigate the problem of minimizing the Average-Value-at-Risk (AV aRr) of the discounted cost over a finite and an infinite horizon which is generated by a Markov Decision Process (MDP). We show that this problem can be reduced to an ordinary MDP with extended state space and give conditions under which an optimal policy exists. We also give a time-consistent interpretation of the AV aRr . At the end we consider a numerical example which is a simple repeated casino game. It is used to discuss the influence of the risk aversion parameter r of the AV aRr-criterion

Markov Decision Processes with Risk-Sensitive Criteria: An Overview

The paper provides an overview of the theory and applications of risk-sensitive Markov decision processes. The term 'risk-sensitive' refers here to the use of the Optimized Certainty Equivalent as a means to measure expectation and risk. This comprises the well-known entropic risk measure and Conditional Value-at-Risk. We restrict our considerations to stationary problems with an infinite time horizon. Conditions are given under which optimal policies exist and solution procedures are explained. We present both the theory when the Optimized Certainty Equivalent is applied recursively as well as the case where it is applied to the cumulated reward. Discounted as well as non-discounted models are reviewe

Minimizing Spectral Risk Measures Applied to Markov Decision Processes

We study the minimization of a spectral risk measure of the total discounted cost generated by a Markov Decision Process (MDP) over a finite or infinite planning horizon. The MDP is assumed to have Borel state and action spaces and the cost function may be unbounded above. The optimization problem is split into two minimization problems using an infimum representation for spectral risk measures. We show that the inner minimization problem can be solved as an ordinary MDP on an extended state space and give sufficient conditions under which an optimal policy exists. Regarding the infinite dimensional outer minimization problem, we prove the existence of a solution and derive an algorithm for its numerical approximation. Our results include the findings in B\"auerle and Ott (2011) in the special case that the risk measure is Expected Shortfall. As an application, we present a dynamic extension of the classical static optimal reinsurance problem, where an insurance company minimizes its cost of capital

Numerical Methods for Random Parameter Optimal Control and the Optimal Control of Stochastic Differential Equations

This thesis considers the investigation and development of numerical methods for optimal control problems that are influenced by stochastic phenomena of various type. The first part treats tasks characterized by random parameters, while in the subsequent second part time-dependent stochastic processes are the basis of the dynamics describing the analyzed systems. In each case the investigations aim to transform the original problem into one that can be tackled by existing (direct) methods of deterministic optimal control - here we prefer Bock's direct multiple shooting approach. In the context of this transformation, in the first part approaches from stochastic programming as well as robust and probabilistic optimization are used. Regarding a specific application from mathematical economics, which considers pricing conspicuous consumption products in periods of recession, new numerical procedures are developed and analyzed with due regard to those techniques - in particular, a scenario tree approach, approximations of robust worst-case settings, and financial tools as the Value at Risk and Conditional Value at Risk. Furthermore, necessary reformulations of the resulting optimal control problems, in particular for Value at Risk and Conditional Value at Risk, as well as the discussion and interpretation of results determined depending on an uncertain recession duration, an uncertain recession strength, and control delays are in focus. The gained economic insight can be seen as an important step in the direction of a better understanding of real-world pricing strategies. In the second part of the thesis, based on the Wiener chaos expansion of a stochastic process and on Malliavin calculus, a system of coupled ordinary differential equations is developed that completely characterizes the stochastic differential equation describing the dynamics of the process. As in general this system includes infinitely many equations, a rigorous error estimation depending on the order of the chaos decomposition is proven in order to guarantee the numerical applicability. To transfer the generic procedure of the chaos expansion to stochastic optimal control problems, a method to preserve the feedback character of the occurring control process is shown. This allows the derivation of a novel direct method to solve finite-horizon stochastic optimal control problems. The appropriability and accuracy of this methodology are demonstrated by treating several problem instances numerically. Finally, the economic application of the first part is revisited under the viewpoint of dealing with a time-dependent recession strength, i.e., a stochastic process. In particular, those applications illustrate that the existing methods of deterministic optimal control can be extended to problems including stochastic differential equations

Preventing premature convergence and proving the optimality in evolutionary algorithms

http://ea2013.inria.fr//proceedings.pdfInternational audienceEvolutionary Algorithms (EA) usually carry out an efficient exploration of the search-space, but get often trapped in local minima and do not prove the optimality of the solution. Interval-based techniques, on the other hand, yield a numerical proof of optimality of the solution. However, they may fail to converge within a reasonable time due to their inability to quickly compute a good approximation of the global minimum and their exponential complexity. The contribution of this paper is a hybrid algorithm called Charibde in which a particular EA, Differential Evolution, cooperates with a Branch and Bound algorithm endowed with interval propagation techniques. It prevents premature convergence toward local optima and outperforms both deterministic and stochastic existing approaches. We demonstrate its efficiency on a benchmark of highly multimodal problems, for which we provide previously unknown global minima and certification of optimality

Untangling hotel industry’s inefficiency: An SFA approach applied to a renowned Portuguese hotel chain

The present paper explores the technical efficiency of four hotels from Teixeira Duarte Group - a renowned Portuguese hotel chain. An efficiency ranking is established from these four hotel units located in Portugal using Stochastic Frontier Analysis. This methodology allows to discriminate between measurement error and systematic inefficiencies in the estimation process enabling to investigate the main inefficiency causes. Several suggestions concerning efficiency improvement are undertaken for each hotel studied.info:eu-repo/semantics/publishedVersio

Modeling and Solving Large-scale Stochastic Mixed-Integer Problems in Transportation and Power Systems

In this dissertation, various optimization problems from the area of transportation and power systems will be respectively investigated and the uncertainty will be considered in each problem. Specifically, a long-term problem of electricity infrastructure investment is studied to address the planning for capacity expansion in electrical power systems with the integration of short-term operations. The future investment costs and real-time customer demands cannot be perfectly forecasted and thus are considered to be random. Another maintenance scheduling problem is studied for power systems, particularly for natural gas fueled power plants, taking into account gas contracting and the opportunity of purchasing and selling gas in the spot market as well as the maintenance scheduling considering the uncertainty of electricity and gas prices in the spot market. In addition, different vehicle routing problems are researched seeking the route for each vehicle so that the total traveling cost is minimized subject to the constraints and uncertain parameters in corresponding transportation systems. The investigation of each problem in this dissertation mainly consists of two parts, i.e., the formulation of its mathematical model and the development of solution algorithm for solving the model. The stochastic programming is applied as the framework to model each problem and address the uncertainty, while the approach of dealing with the randomness varies in terms of the relationships between the uncertain elements and objective functions or constraints. All the problems will be modeled as stochastic mixed-integer programs, and the huge numbers of involved decision variables and constraints make each problem large-scale and very difficult to manage. In this dissertation, efficient algorithms are developed for these problems in the context of advanced methodologies of optimization and operations research, such as branch and cut, benders decomposition, column generation and Lagrangian method. Computational experiments are implemented for each problem and the results will be present and discussed. The research carried out in this dissertation would be beneficial to both researchers and practitioners seeking to model and solve similar optimization problems in transportation and power systems when uncertainty is involved

Software reliability modeling and release time determination

Ph.DDOCTOR OF PHILOSOPH