Approximate dynamic programming for anemia management.

The focus of this dissertation work is the formulation and improvement of anemia management process involving trial-and-error. A two-stage method is adopted toward this objective. Given a medical treatment process, a discrete Markov representation is first derived as a formal translation of the treatment process to a control problem under uncertainty. A simulative numerical solution of the control problem is then obtained on-the-fly in the form of a control law maximizing the long-term benefit at each decision stage. Approximate dynamic programming methods are employed in the proposed solution. The motivation underlying this choice is that, in reality, some patient characteristics, which are critical for the sake of treatment, cannot be determined through diagnosis and remain unknown until early stages of treatment, when the patient demonstrates them upon actions by the decision maker. A review of these simulative control tools, which are studied extensively in reinforcement learning theory, is presented. Two approximate dynamic programming tools, namely SARSA and Q -learning, are introduced. Their performance in discovering the optimal individualized drug dosing policy is illustrated on hypothetical patients made up as fuzzy models for simulations. As an addition to these generic reinforcement learning methods, a state abstraction scheme for the considered application domain is also proposed. The control methods of this study, capturing the essentials of a drug delivery problem, constitutes a novel computational framework for model-free medical treatment. Experimental evaluation of the dosing strategies produced by the proposed methods against the standard policy, which is being followed actually by human experts in Kidney Diseases Program, University of Louisville, shows the advantages for use of reinforcement learning in the drug dosing problem in particular and in medical decision making in general

Cops and Invisible Robbers: the Cost of Drunkenness

We examine a version of the Cops and Robber (CR) game in which the robber is invisible, i.e., the cops do not know his location until they capture him. Apparently this game (CiR) has received little attention in the CR literature. We examine two variants: in the first the robber is adversarial (he actively tries to avoid capture); in the second he is drunk (he performs a random walk). Our goal in this paper is to study the invisible Cost of Drunkenness (iCOD), which is defined as the ratio ct_i(G)/dct_i(G), with ct_i(G) and dct_i(G) being the expected capture times in the adversarial and drunk CiR variants, respectively. We show that these capture times are well defined, using game theory for the adversarial case and partially observable Markov decision processes (POMDP) for the drunk case. We give exact asymptotic values of iCOD for several special graph families such as $d$-regular trees, give some bounds for grids, and provide general upper and lower bounds for general classes of graphs. We also give an infinite family of graphs showing that iCOD can be arbitrarily close to any value in [2,infinty). Finally, we briefly examine one more CiR variant, in which the robber is invisible and "infinitely fast"; we argue that this variant is significantly different from the Graph Search game, despite several similarities between the two games

An intelligent resource allocation decision support system with Q-learning

Master'sMASTER OF ENGINEERIN

Advances in Reinforcement Learning

Reinforcement Learning (RL) is a very dynamic area in terms of theory and application. This book brings together many different aspects of the current research on several fields associated to RL which has been growing rapidly, producing a wide variety of learning algorithms for different applications. Based on 24 Chapters, it covers a very broad variety of topics in RL and their application in autonomous systems. A set of chapters in this book provide a general overview of RL while other chapters focus mostly on the applications of RL paradigms: Game Theory, Multi-Agent Theory, Robotic, Networking Technologies, Vehicular Navigation, Medicine and Industrial Logistic

Computational intelligence approaches to robotics, automation, and control [Volume guest editors]

No abstract available

Reinforcement Learning

Brains rule the world, and brain-like computation is increasingly used in computers and electronic devices. Brain-like computation is about processing and interpreting data or directly putting forward and performing actions. Learning is a very important aspect. This book is on reinforcement learning which involves performing actions to achieve a goal. The first 11 chapters of this book describe and extend the scope of reinforcement learning. The remaining 11 chapters show that there is already wide usage in numerous fields. Reinforcement learning can tackle control tasks that are too complex for traditional, hand-designed, non-learning controllers. As learning computers can deal with technical complexities, the tasks of human operators remain to specify goals on increasingly higher levels. This book shows that reinforcement learning is a very dynamic area in terms of theory and applications and it shall stimulate and encourage new research in this field

Stochastic optimal adaptive controller and communication protocol design for networked control systems

Networked Control System (NCS) is a recent topic of research wherein the feedback control loops are closed through a real-time communication network. Many design challenges surface in such systems due to network imperfections such as random delays, packet losses, quantization effects and so on. Since existing control techniques are unsuitable for such systems, in this dissertation, a suite of novel stochastic optimal adaptive design methodologies is undertaken for both linear and nonlinear NCS in presence of uncertain system dynamics and unknown network imperfections such as network-induced delays and packet losses. The design is introduced in five papers. In Paper 1, a stochastic optimal adaptive control design is developed for unknown linear NCS with uncertain system dynamics and unknown network imperfections. A value function is adjusted forward-in-time and online, and a novel update law is proposed for tuning value function estimator parameters. Additionally, by using estimated value function, optimal adaptive control law is derived based on adaptive dynamic programming technique. Subsequently, this design methodology is extended to solve stochastic optimal strategies of linear NCS zero-sum games in Paper 2. Since most systems are inherently nonlinear, a novel stochastic optimal adaptive control scheme is then developed in Paper 3 for nonlinear NCS with unknown network imperfections. On the other hand, in Paper 4, the network protocol behavior (e.g. TCP and UDP) are considered and optimal adaptive control design is revisited using output feedback for linear NCS. Finally, Paper 5 explores a co-design framework where both the controller and network scheduling protocol designs are addressed jointly so that proposed scheme can be implemented into next generation Cyber Physical Systems --Abstract, page iv

Machine Learning

Machine Learning can be defined in various ways related to a scientific domain concerned with the design and development of theoretical and implementation tools that allow building systems with some Human Like intelligent behavior. Machine learning addresses more specifically the ability to improve automatically through experience

A Comprehensive Review of Bio-Inspired Optimization Algorithms Including Applications in Microelectronics and Nanophotonics

The application of artificial intelligence in everyday life is becoming all-pervasive and unavoidable. Within that vast field, a special place belongs to biomimetic/bio-inspired algorithms for multiparameter optimization, which find their use in a large number of areas. Novel methods and advances are being published at an accelerated pace. Because of that, in spite of the fact that there are a lot of surveys and reviews in the field, they quickly become dated. Thus, it is of importance to keep pace with the current developments. In this review, we first consider a possible classification of bio-inspired multiparameter optimization methods because papers dedicated to that area are relatively scarce and often contradictory. We proceed by describing in some detail some more prominent approaches, as well as those most recently published. Finally, we consider the use of biomimetic algorithms in two related wide fields, namely microelectronics (including circuit design optimization) and nanophotonics (including inverse design of structures such as photonic crystals, nanoplasmonic configurations and metamaterials). We attempted to keep this broad survey self-contained so it can be of use not only to scholars in the related fields, but also to all those interested in the latest developments in this attractive area