Inventory control for point-of-use locations in hospitals

Most inventory management systems at hospital departments are characterised by lost sales, periodic reviews with short lead times, and limited storage capacity. We develop two types of exact models that deal with all these characteristics. In a capacity model, the service level is maximised subject to a capacity restriction, and in a service model the required capacity is minimised subject to a service level restriction. We also formulate approximation models applicable for any lost-sales inventory system (cost objective, no lead time restrictions etc). For the capacity model, we develop a simple inventory rule to set the reorder levels and order quantities. Numerical results for this inventory rule show an average deviation of 1% from the optimal service levels. We also embed the single-item models in a multi-item system. Furthermore, we compare the performance of fixed order size replenishment policies and (R, s, S) policies

New prioritized value iteration for Markov decision processes

The problem of solving large Markov decision processes accurately and quickly is challenging. Since the computational effort incurred is considerable, current research focuses on finding superior acceleration techniques. For instance, the convergence properties of current solution methods depend, to a great extent, on the order of backup operations. On one hand, algorithms such as topological sorting are able to find good orderings but their overhead is usually high. On the other hand, shortest path methods, such as Dijkstra's algorithm which is based on priority queues, have been applied successfully to the solution of deterministic shortest-path Markov decision processes. Here, we propose an improved value iteration algorithm based on Dijkstra's algorithm for solving shortest path Markov decision processes. The experimental results on a stochastic shortest-path problem show the feasibility of our approach. © Springer Science+Business Media B.V. 2011.García Hernández, MDG.; Ruiz Pinales, J.; Onaindia De La Rivaherrera, E.; Aviña Cervantes, JG.; Ledesma Orozco, S.; Alvarado Mendez, E.; Reyes Ballesteros, A. (2012). New prioritized value iteration for Markov decision processes. Artificial Intelligence Review. 37(2):157-167. doi:10.1007/s10462-011-9224-zS157167372Agrawal S, Roth D (2002) Learning a sparse representation for object detection. In: Proceedings of the 7th European conference on computer vision. Copenhagen, Denmark, pp 1–15Bellman RE (1954) The theory of dynamic programming. Bull Amer Math Soc 60: 503–516Bellman RE (1957) Dynamic programming. Princeton University Press, New JerseyBertsekas DP (1995) Dynamic programming and optimal control. Athena Scientific, MassachusettsBhuma K, Goldsmith J (2003) Bidirectional LAO* algorithm. In: Proceedings of indian international conferences on artificial intelligence. p 980–992Blackwell D (1965) Discounted dynamic programming. Ann Math Stat 36: 226–235Bonet B, Geffner H (2003a) Faster heuristic search algorithms for planning with uncertainty and full feedback. In: Proceedings of the 18th international joint conference on artificial intelligence. Morgan Kaufmann, Acapulco, México, pp 1233–1238Bonet B, Geffner H (2003b) Labeled RTDP: improving the convergence of real-time dynamic programming. In: Proceedings of the international conference on automated planning and scheduling. Trento, Italy, pp 12–21Bonet B, Geffner H (2006) Learning depth-first search: a unified approach to heuristic search in deterministic and non-deterministic settings and its application to MDP. In: Proceedings of the 16th international conference on automated planning and scheduling. Cumbria, UKBoutilier C, Dean T, Hanks S (1999) Decision-theoretic planning: structural assumptions and computational leverage. J Artif Intell Res 11: 1–94Chang I, Soo H (2007) Simulation-based algorithms for Markov decision processes Communications and control engineering. Springer, LondonDai P, Goldsmith J (2007a) Faster dynamic programming for Markov decision processes. Technical report. Doctoral consortium, department of computer science and engineering. University of WashingtonDai P, Goldsmith J (2007b) Topological value iteration algorithm for Markov decision processes. In: Proceedings of the 20th international joint conference on artificial intelligence. Hyderabad, India, pp 1860–1865Dai P, Hansen EA (2007c) Prioritizing bellman backups without a priority queue. In: Proceedings of the 17th international conference on automated planning and scheduling, association for the advancement of artificial intelligence. Rhode Island, USA, pp 113–119Dibangoye JS, Chaib-draa B, Mouaddib A (2008) A Novel prioritization technique for solving Markov decision processes. In: Proceedings of the 21st international FLAIRS (The Florida Artificial Intelligence Research Society) conference, association for the advancement of artificial intelligence. Florida, USAFerguson D, Stentz A (2004) Focused propagation of MDPs for path planning. In: Proceedings of the 16th IEEE international conference on tools with artificial intelligence. pp 310–317Hansen EA, Zilberstein S (2001) LAO: a heuristic search algorithm that finds solutions with loops. Artif Intell 129: 35–62Hinderer K, Waldmann KH (2003) The critical discount factor for finite Markovian decision processes with an absorbing set. Math Methods Oper Res 57: 1–19Li L (2009) A unifying framework for computational reinforcement learning theory. PhD Thesis. The state university of New Jersey, New Brunswick. NJLittman ML, Dean TL, Kaelbling LP (1995) On the complexity of solving Markov decision problems.In: Proceedings of the 11th international conference on uncertainty in artificial intelligence. Montreal, Quebec pp 394–402McMahan HB, Gordon G (2005a) Fast exact planning in Markov decision processes. In: Proceedings of the 15th international conference on automated planning and scheduling. Monterey, CA, USAMcMahan HB, Gordon G (2005b) Generalizing Dijkstra’s algorithm and gaussian elimination for solving MDPs. Technical report, Carnegie Mellon University, PittsburghMeuleau N, Brafman R, Benazera E (2006) Stochastic over-subscription planning using hierarchies of MDPs. In: Proceedings of the 16th international conference on automated planning and scheduling. Cumbria, UK, pp 121–130Moore A, Atkeson C (1993) Prioritized sweeping: reinforcement learning with less data and less real time. Mach Learn 13: 103–130Puterman ML (1994) Markov decision processes. Wiley Editors, New YorkPuterman ML (2005) Markov decision processes. Wiley Inter Science Editors, New YorkRussell S (2005) Artificial intelligence: a modern approach. Making complex decisions (Ch-17), 2nd edn. Pearson Prentice Hill Ed., USAShani G, Brafman R, Shimony S (2008) Prioritizing point-based POMDP solvers. IEEE Trans Syst Man Cybern 38(6): 1592–1605Sniedovich M (2006) Dijkstra’s algorithm revisited: the dynamic programming connexion. Control Cybern 35: 599–620Sniedovich M (2010) Dynamic programming: foundations and principles, 2nd edn. Pure and Applied Mathematics Series, UKTijms HC (2003) A first course in stochastic models. Discrete-time Markov decision processes (Ch-6). Wiley Editors, UKVanderbei RJ (1996) Optimal sailing strategies. Statistics and operations research program, University of Princeton, USA ( http://www.orfe.princeton.edu/~rvdb/sail/sail.html )Vanderbei RJ (2008) Linear programming: foundations and extensions, 3rd edn. Springer, New YorkWingate D, Seppi KD (2005) Prioritization methods for accelerating MDP solvers. J Mach Learn Res 6: 851–88

Analysis of finite-buffer state-dependent bulk queues

<p>In this paper, we consider a general state-dependent finite-buffer bulk queue in which the rates and batch sizes of arrivals and services are allowed to depend on the number of customers in queue and service batch sizes. Such queueing systems have rich applications in manufacturing, service operations, computer and telecommunication systems. Interesting examples include batch oven processes in the aircraft and semiconductor industry; serving of passengers by elevators, shuttle buses, and ferries; and congestion control mechanisms to regulate transmission rates in packet-switched communication networks. We develop a unifying method to study the performance of this general class of finite-buffer state-dependent bulk queueing systems. For this purpose, we use semi-regenerative analysis to develop a numerically stable method for calculating the limiting probability distribution of the queue length process. Based on the limiting probabilities, we present various performance measures for evaluating admission control and batch service policies, such as the loss probability for an arriving group of customers and for individual customers within a group. We demonstrate our method by means of numerical examples.</p>

Performance evaluation of scheduling policies for the DRCMPSP

In this study, we consider the dynamic resource-constrained multi-project scheduling problem (DRCMPSP) where projects generate rewards at their completion, completions later than a due date cause tardiness costs and new projects arrive randomly during the ongoing project execution which disturbs the existing project scheduling plan. We model this problem as a discrete Markov decision process and explore the computational limitations of solving the problem by dynamic programming. We run and compare four different solution approaches on small size problems. These solution approaches are: a dynamic programming algorithm to determine a policy that maximises the average profit per unit time net of charges for late project completion, a genetic algorithm which generates a schedule to maximise the total reward of ongoing projects and updates the schedule with each new project arrival, a rule-based algorithm which prioritise processing of tasks with the highest processing durations, and a worst decision algorithm to seek a non-idling policy to minimise the average profit per unit time. Average profits per unit time of generated policies of the solution algorithms are evaluated and compared. The performance of the genetic algorithm is the closest to the optimal policies of the dynamic programming algorithm, but its results are notably suboptimal, up to 67.2\%. Alternative scheduling algorithms are close to optimal with low project arrival probability but quickly deteriorate their performance as the probability increases

Iterative approximation of k-limited polling systems

The present paper deals with the problem of calculating queue length distributions in a polling model with (exhaustive) k-limited service under the assumption of general arrival, service and setup distributions. The interest for this model is fueled by an application in the field of logistics. Knowledge of the queue length distributions is needed to operate the system properly. The multi-queue polling system is decomposed into single-queue vacation systems with k-limited service and state-dependent vacations, for which the vacation distributions are computed in an iterative approximate manner. These vacation models are analyzed via matrix-analytic techniques. The accuracy of the approximation scheme is verified by means of an extensive simulation study. The developed approximation turns out be accurate, robust and computationally efficient

A Spoonful of Math Helps the Medicine Go Down: An Illustration of How Healthcare can Benefit from Mathematical Modeling and Analysis

<p>Abstract</p> <p>Objectives</p> <p>A recent joint report from the Institute of Medicine and the National Academy of Engineering, highlights the benefits of--indeed, the need for--mathematical analysis of healthcare delivery. Tools for such analysis have been developed over decades by researchers in Operations Research (OR). An OR perspective typically frames a complex problem in terms of its essential mathematical structure. This article illustrates the use and value of the tools of operations research in healthcare. It reviews one OR tool, queueing theory, and provides an illustration involving a hypothetical drug treatment facility.</p> <p>Method</p> <p>Queueing Theory (QT) is the study of waiting lines. The theory is useful in that it provides solutions to problems of waiting and its relationship to key characteristics of healthcare systems. More generally, it illustrates the strengths of modeling in healthcare and service delivery.</p> <p>Queueing theory offers insights that initially may be hidden. For example, a queueing model allows one to incorporate randomness, which is inherent in the actual system, into the mathematical analysis. As a result of this randomness, these systems often perform much worse than one might have guessed based on deterministic conditions. Poor performance is reflected in longer lines, longer waits, and lower levels of server utilization.</p> <p>As an illustration, we specify a queueing model of a representative drug treatment facility. The analysis of this model provides mathematical expressions for some of the key performance measures, such as average waiting time for admission.</p> <p>Results</p> <p>We calculate average occupancy in the facility and its relationship to system characteristics. For example, when the facility has 28 beds, the average wait for admission is 4 days. We also explore the relationship between arrival rate at the facility, the capacity of the facility, and waiting times.</p> <p>Conclusions</p> <p>One key aspect of the healthcare system is its complexity, and policy makers want to design and reform the system in a way that affects competing goals. OR methodologies, particularly queueing theory, can be very useful in gaining deeper understanding of this complexity and exploring the potential effects of proposed changes on the system without making any actual changes.</p