Analysis and Simplex-type Algorithms for Countably Infinite Linear Programming Models of Markov Decision Processes.

The class of Markov decision processes (MDPs) provides a popular framework which covers a wide variety of sequential decision-making problems. We consider infinite-horizon discounted MDPs with countably infinite state space and finite action space. Our goal is to establish theoretical properties and develop new solution methods for such MDPs by studying their linear programming (LP) formulations. The LP formulations have countably infinite numbers of variables and constraints and therefore are called countably infinite linear programs (CILPs). General CILPs are challenging to analyze or solve, mainly because useful theoretical properties and techniques of finite LPs fail to extend to general CILPs. Another goal of this thesis is to deepen the limited current understanding of CILPs, resulting in new algorithmic approaches to find their solutions. Recently, Ghate and Smith (2013) developed an implementable simplex-type algorithm for solving a CILP formulation of a non-stationary MDP with finite state space. We establish rate of convergence results for their simplex algorithm with a particular pivoting rule and another existing solution method for such MDPs, and compare empirical performance of the algorithms. We also present ways to accelerate their simplex algorithm. The class of non-stationary MDPs with finite state space can be considered to be a subclass of stationary MDPs with countably infinite state space. We present a simplex-type algorithm for solving a CILP formulation of a stationary MDP with countably infinite state space that is implementable (using only finite data and computation in each iteration). We show that the algorithm finds a sequence of policies that improves monotonically and converges to optimality in value, and present a numerical illustration. An important extension of MDPs considered so far are constrained MDPs, which optimize an objective function while satisfying constraints, typically on budget, quality, and so on. For constrained non-stationary MDPs with finite state space, we provide a necessary and sufficient condition for a feasible solution of its CILP formulation to be an extreme point. Since simplex-type algorithms are expected to navigate between extreme points, this result sets a foundation for developing a simplex-type algorithm for constrained non-stationary MDPs.PhDIndustrial and Operations EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/113486/1/ilbinlee_1.pd

Discrete-time controlled markov processes with average cost criterion: a survey

This work is a survey of the average cost control problem for discrete-time Markov processes. The authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades. The exposition ranges from finite to Borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies. The authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography. The authors have also identified several important questions that are still open to investigation

Simplex Algorithm for Countable-state Discounted Markov Decision Processes

Submitted to Operations Research; preliminary version.We consider discounted Markov Decision Processes (MDPs) with countably-infinite state spaces, finite action spaces, and unbounded rewards. Typical examples of such MDPs are inventory management and queueing control problems in which there is no specific limit on the size of inventory or queue. Existing solution methods obtain a sequence of policies that converges to optimality in value but may not improve monotonically, i.e., a policy in the sequence may be worse than preceding policies. Our proposed approach considers countably-infinite linear programming (CILP) formulations of the MDPs (a CILP is defined as a linear program (LP) with countably-infinite numbers of variables and constraints). Under standard assumptions for analyzing MDPs with countably-infinite state spaces and unbounded rewards, we extend the major theoretical extreme point and duality results to the resulting CILPs. Under an additional technical assumption which is satisfied by several applications of interest, we present a simplex-type algorithm that is implementable in the sense that each of its iterations requires only a finite amount of data and computation. We show that the algorithm finds a sequence of policies which improves monotonically and converges to optimality in value. Unlike existing simplex-type algorithms for CILPs, our proposed algorithm solves a class of CILPs in which each constraint may contain an infinite number of variables and each variable may appear in an infinite number of constraints. A numerical illustration for inventory management problems is also presented.National Science Foundation grant CMMI-1333260A research grant from the University of Michiganhttp://deepblue.lib.umich.edu/bitstream/2027.42/109413/1/CountableStateMDP-MAE.pdfDescription of CountableStateMDP-MAE.pdf : Main article (preliminary version

Modeling Uncertainty in Large Natural Resource Allocation Problems

The productivity of the world's natural resources is critically dependent on a variety of highly uncertain factors, which obscure individual investors and governments that seek to make long-term, sometimes irreversible investments in their exploration and utilization. These dynamic considerations are poorly represented in disaggregated resource models, as incorporating uncertainty into large-dimensional problems presents a challenging computational task. This study introduces a novel numerical method to solve large-scale dynamic stochastic natural resource allocation problems that cannot be addressed by conventional methods. The method is illustrated with an application focusing on the allocation of global land resource use under stochastic crop yields due to adverse climate impacts and limits on further technological progress. For the same model parameters, the range of land conversion is considerably smaller for the dynamic stochastic model as compared to deterministic scenario analysis. The scenario analysis can thus significantly overstate the magnitude of expected land conversion under uncertain crop yields

Bargaining with Incomplete Information

A central question in economics is understanding the difficulties that parties have in reaching mutually beneficial agreements. Informational differences provide an appealing explanation for bargaining inefficiencies. This chapter provides an overview of the theoretical and empirical literature on bargaining with incomplete information. The chapter begins with an analysis of bargaining within a mechanism design framework. A modern development is provided of the classic result that, given two parties with independent private valuations, ex post efficiency is attainable if and only if it is common knowledge that gains from trade exist. The classic problems of efficient trade with one-sided incomplete information but interdependent valuations, and of efficiently dissolving a partnership with two-sided incomplete information, are also reviewed using mechanism design. The chapter then proceeds to study bargaining where the parties sequentially exchange offers. Under one-sided incomplete information, it considers sequential bargaining between a seller with a known valuation and a buyer with a private valuation. When there is a "gap" between the seller's valuation and the support of buyer valuations, the seller-offer game has essentially a unique sequential equilibrium. This equilibrium exhibits the following properties: it is stationary, trade occurs in finite time, and the price is favorable to the informed party (the Coase Conjecture). The alternating-offer game exhibits similar properties, when a refinement of sequential equilibrium is applied. However, in the case of "no gap" between the seller's valuation and the support of buyer valuations, the bargaining does not conclude with probability one after any finite number of periods, and it does not follow that sequential equilibria need be stationary. If stationarity is nevertheless assumed, then the results parallel those for the "gap" case. However, if stationarity is not assumed, then instead a folk theorem obtains, so substantial delay is possible and the uninformed party may receive substantial surplus. The chapter also briefly sketches results for sequential bargaining with two-sided incomplete information. Finally, it reviews the empirical evidence on strategic bargaining with private information by focusing on one of the most prominent examples of bargaining: union contract negotiations.Bargaining; Delay; Incomplete Information

Motivational Ratings

Performance evaluation (\rating") systems not only provide information to users but also motivate the rated worker. This paper solves for the optimal(effort-maximizing) rating within the standard career concerns framework. We prove that this rating is a linear function of past observations. The rating, however, is not a Markov process, but rather the sum of two Markov processes. We show how it combines information of different types and vintages. An increase in effort may adversely affect some (but not all) future ratings

Data-driven reconfigurable supply chain design and inventory control

In this dissertation, we examine resource mobility in a supply chain that attempts to satisfy geographically distributed demand through resource sharing, where the resources can be inventory and manufacturing capacity. Our objective is to examine how resource mobility, coupled with data-driven analytics, can result in supply chains that without customer service level reduction blend the advantages of distributed production-inventory systems (e.g., fast fulfillment) and centralized systems (e.g., economies of scale, less total buffer inventory, and reduced capital expenditures). We present efficient and effective solution methods for logistics management of multi-location production-inventory systems with transportable production capacity. We present a novel, generalized representation of demand uncertainty and propose data-driven responses to the manage a single location inventory system under such demands.Ph.D