Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

The exponential cost optimality for finite horizon semi-Markov decision processes

summary:This paper considers an exponential cost optimality problem for finite horizon semi-Markov decision processes (SMDPs). The objective is to calculate an optimal policy with minimal exponential costs over the full set of policies in a finite horizon. First, under the standard regular and compact-continuity conditions, we establish the optimality equation, prove that the value function is the unique solution of the optimality equation and the existence of an optimal policy by using the minimum nonnegative solution approach. Second, we establish a new value iteration algorithm to calculate both the value function and the $\epsilon$-optimal policy. Finally, we give a computable machine maintenance system to illustrate the convergence of the algorithm

Maintenance optimization of a production system with buffercapacity

Marketing;Optimization;produktieleer/ produktieplanning

A unified methodology of maintenance management for repairable systems based on optimal stopping theory

This dissertation focuses on the study of maintenance management for repairable systems based on optimal stopping theory. From reliability engineering’s point of view, all systems are subject to deterioration with age and usage. System deterioration can take various forms, including wear, fatigue, fracture, cracking, breaking, corrosion, erosion and instability, any of which may ultimately cause the system to fail to perform its required function. Consequently, controlling system deterioration through maintenance and thus controlling the risk of system failure becomes beneficial or even necessary. Decision makers constantly face two fundamental problems with respect to system maintenance. One is whether or when preventive maintenance should be performed in order to avoid costly failures. The other problem is how to make the choice among different maintenance actions in response to a system failure. The whole purpose of maintenance management is to keep the system in good working condition at a reasonably low cost, thus the tradeoff between cost and condition plays a central role in the study of maintenance management, which demands rigorous optimization. The agenda of this research is to develop a unified methodology for modeling and optimization of maintenance systems. A general modeling framework with six classifying criteria is to be developed to formulate and analyze a wide range of maintenance systems which include many existing models in the literature. A unified optimization procedure is developed based on optimal stopping, semi-martingale, and lambda-maximization techniques to solve these models contained in the framework. A comprehensive model is proposed and solved in this general framework using the developed procedure which incorporates many other models as special cases. Policy comparison and policy optimality are studied to offer further insights. Along the theoretical development, numerical examples are provided to illustrate the applicability of the methodology. The main contribution of this research is that the unified modeling framework and systematic optimization procedure structurize the pool of models and policies, weed out non-optimal policies, and establish a theoretical foundation for further development

Pilot interaction with automated airborne decision making systems

An investigation was made of interaction between a human pilot and automated on-board decision making systems. Research was initiated on the topic of pilot problem solving in automated and semi-automated flight management systems and attempts were made to develop a model of human decision making in a multi-task situation. A study was made of allocation of responsibility between human and computer, and discussed were various pilot performance parameters with varying degrees of automation. Optimal allocation of responsibility between human and computer was considered and some theoretical results found in the literature were presented. The pilot as a problem solver was discussed. Finally the design of displays, controls, procedures, and computer aids for problem solving tasks in automated and semi-automated systems was considered

On control of discrete-time state-dependent jump linear systems with probabilistic constraints: A receding horizon approach

In this article, we consider a receding horizon control of discrete-time state-dependent jump linear systems, particular kind of stochastic switching systems, subject to possibly unbounded random disturbances and probabilistic state constraints. Due to a nature of the dynamical system and the constraints, we consider a one-step receding horizon. Using inverse cumulative distribution function, we convert the probabilistic state constraints to deterministic constraints, and obtain a tractable deterministic receding horizon control problem. We consider the receding control law to have a linear state-feedback and an admissible offset term. We ensure mean square boundedness of the state variable via solving linear matrix inequalities off-line, and solve the receding horizon control problem on-line with control offset terms. We illustrate the overall approach applied on a macroeconomic system