Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

A multilevel integrative approach to hospital case mix and capacity planning.

Hospital case mix and capacity planning involves the decision making both on patient volumes that can be taken care of at a hospital and on resource requirements and capacity management. In this research, to advance both the hospital resource efficiency and the health care service level, a multilevel integrative approach to the planning problem is proposed on the basis of mathematical programming modeling and simulation analysis. It consists of three stages, namely the case mix planning phase, the master surgery scheduling phase and the operational performance evaluation phase. At the case mix planning phase, a hospital is assumed to choose the optimal patient mix and volume that can bring the maximum overall financial contribution under the given resource capacity. Then, in order to improve the patient service level potentially, the total expected bed shortage due to the variable length of stay of patients is minimized through reallocating the bed capacity and building balanced master surgery schedules at the master surgery scheduling phase. After that, the performance evaluation is carried out at the operational stage through simulation analysis, and a few effective operational policies are suggested and analyzed to enhance the trade-offs between resource efficiency and service level. The three stages are interacting and are combined in an iterative way to make sound decisions both on the patient case mix and on the resource allocation.Health care; Case mix and capacity planning; Master surgery schedule; Multilevel; Resource efficiency; Service level;

A Case Study in Optimization of Resource Distribution to Cope with Unanticipated Changes in Requirements

It is a known fact that requirements change continuously, and as a consequence, it may be necessary to reschedule development activities so that the new requirements can be addressed in a costeffective manner. Unfortunately, changes in requirements cannot be specified precisely. Moreover, current software development methods do not provide explicit means to adapt development processes with respect to unanticipated changes in requirements. This article first proposes a method based on Markov Decision Theory, which determines the estimated optimal development schedule with respect to probabilistic product demands and resource constraints. Second, a tool is described that is built to support the method. Finally, some experimental results are presented on the applicability of the proposed method

Human-Machine Collaborative Optimization via Apprenticeship Scheduling

Coordinating agents to complete a set of tasks with intercoupled temporal and resource constraints is computationally challenging, yet human domain experts can solve these difficult scheduling problems using paradigms learned through years of apprenticeship. A process for manually codifying this domain knowledge within a computational framework is necessary to scale beyond the ``single-expert, single-trainee" apprenticeship model. However, human domain experts often have difficulty describing their decision-making processes, causing the codification of this knowledge to become laborious. We propose a new approach for capturing domain-expert heuristics through a pairwise ranking formulation. Our approach is model-free and does not require enumerating or iterating through a large state space. We empirically demonstrate that this approach accurately learns multifaceted heuristics on a synthetic data set incorporating job-shop scheduling and vehicle routing problems, as well as on two real-world data sets consisting of demonstrations of experts solving a weapon-to-target assignment problem and a hospital resource allocation problem. We also demonstrate that policies learned from human scheduling demonstration via apprenticeship learning can substantially improve the efficiency of a branch-and-bound search for an optimal schedule. We employ this human-machine collaborative optimization technique on a variant of the weapon-to-target assignment problem. We demonstrate that this technique generates solutions substantially superior to those produced by human domain experts at a rate up to 9.5 times faster than an optimization approach and can be applied to optimally solve problems twice as complex as those solved by a human demonstrator.Comment: Portions of this paper were published in the Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) in 2016 and in the Proceedings of Robotics: Science and Systems (RSS) in 2016. The paper consists of 50 pages with 11 figures and 4 table

Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks

Future wireless networks have a substantial potential in terms of supporting a broad range of complex compelling applications both in military and civilian fields, where the users are able to enjoy high-rate, low-latency, low-cost and reliable information services. Achieving this ambitious goal requires new radio techniques for adaptive learning and intelligent decision making because of the complex heterogeneous nature of the network structures and wireless services. Machine learning (ML) algorithms have great success in supporting big data analytics, efficient parameter estimation and interactive decision making. Hence, in this article, we review the thirty-year history of ML by elaborating on supervised learning, unsupervised learning, reinforcement learning and deep learning. Furthermore, we investigate their employment in the compelling applications of wireless networks, including heterogeneous networks (HetNets), cognitive radios (CR), Internet of things (IoT), machine to machine networks (M2M), and so on. This article aims for assisting the readers in clarifying the motivation and methodology of the various ML algorithms, so as to invoke them for hitherto unexplored services as well as scenarios of future wireless networks.Comment: 46 pages, 22 fig

Gestión logística de sistemas de hospitalización domiciliaria: una revisión crítica de modelos y métodos

RESUMEN: Los servicios de Hospitalización Domiciliaria (HD) se basan en una red de distribución, en la cual los pacientes son hospitalizados en sus casas y los prestadores de servicios de salud deben entregar cuidados médicos coordinados a los pacientes. La demanda de estos servicios está creciendo rápidamente y los gobiernos y proveedores de servicios de salud enfrentan el reto de tomar un conjunto de decisiones complejas en un sector con un componente logístico importante. En este artículo se presenta una revisión crítica de los modelos y métodos utilizados para darle soporte a las decisiones logísticas en HD. Para esto se presenta primero un marco de referencia, con el objetivo de identificar las oportunidades de investigación en el campo. Con base en dicho marco, se presenta la revisión de la literatura y la identificación de brechas en la investigación. En particular, se hace énfasis en la necesidad de desarrollar e implementar metodologías más integradas para dar soporte a las decisiones estratégicas y tácticas y de considerar puntos clave de los sistemas reales.ABSTRACT: Home Health Care (HHC) services are based on a delivery network in which patients are hospitalized at their homes and health care providers must deliver coordinated medical care to patients. Demand for HHC services is rapidly growing and governments and health care providers face the challenge to make a set of complex decisions in a medical service business that has an important component of logistics problems. The objective of this paper is to provide a critical review of models and methods used to support logistics decisions in HHC. For this purpose, a reference framework is proposed first in order to identify research perspectives in the field. Based on this framework, a literature review is presented and research gaps are identified. In particular, the literature review reveals that more emphasizes is needed to develop and implement more integrated methodologies to support decisions at tactical and strategic planning levels and to consider key features from real systems