Reinforcement Learning: A Survey

This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word ``reinforcement.'' The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.Comment: See http://www.jair.org/ for any accompanying file

Learning algorithms for adaptive digital filtering

In this thesis, we consider the problem of parameter optimisation in adaptive digital filtering. Adaptive digital filtering can be accomplished using both Finite Impulse Response (FIR) filters and Infinite Impulse Response Filters (IIR) filters. Adaptive FIR filtering algorithms are well established. However, the potential computational advantages of IIR filters has led to an increase in research on adaptive IIR filtering algorithms. These algorithms are studied in detail in this thesis and the limitations of current adaptive IIR filtering algorithms are identified. New approaches to adaptive IIR filtering using intelligent learning algorithms are proposed. These include Stochastic Learning Automata, Evolutionary Algorithms and Annealing Algorithms. Each of these techniques are used for the filtering problem and simulation results are presented showing the performance of the algorithms for adaptive IIR filtering. The relative merits and demerits of the different schemes are discussed. Two practical applications of adaptive IIR filtering are simulated and results of using the new adaptive strategies are presented. Other than the new approaches used, two new hybrid schemes are proposed based on concepts from genetic algorithms and annealing. It is shown with the help of simulation studies, that these hybrid schemes provide a superior performance to the exclusive use of any one scheme

Theory and application of learning automata.

Although the theoretical performance of many learning automata has been considered, the practical operation of these automata has received far less attention. This work starts with the construction of two action Tsetlin and Krylov automata. The performance of these automata has been measured in stationary and non-stationary environments. The operation of a hierarchical automaton controlling the memory size of a Tsetlin automaton is also investigated. Two new automata are proposed with the aim of avoiding the operational disadvantages of the Tsetlin automaton. These automata have been tested using a computer simulation and, in addition, theoretical performance results have been calculated and compared with results for Tsetlin, Krylov and Lri automata. A model of a non-autonomous environment is simulated and its operation analysed theoretically. A more accurate model is analysed, and its operation with a Lri automaton examined and compared to theoretical predictions. The requirements for learning automata to operate successfully in non-autonomous environments are considered and it is shown that the Lrp and Lri automata do not converge to the optimum for a non-autonomous environment. Three automata are proposed, which are designed to operate in non-autonomous environments. Their performances are compared to those of the Lrp and Lri automata. The operation of automata in a hierarchical learning system and in cooperative and competitive games is considered. In these situations the performance of the new automata is compared to that of the Lrp and Lri automata. Finally, two applications of learning automata are investigated. The first considers the Tsetlin allocation scheme, gives a modification that increases the performance and makes a comparison with a scheme using other learning automata. The second involves the selection of a processor in a multiprocessor computer system and compares a scheme using learning automata with a fixed scheduling discipline

Study of decentralised decision models in distributed environments

Many of today's complex systems require effective decision making within uncertain distributed environments. The central theme of the thesis considers the systematic analysis for the representation of decision making organisations. The basic concept of stochastic learning automata provides a framework for modelling decision making in complex systems. Models of interactive decision making are discussed, which result from interconnecting decision makers in both synchronous and sequential configurations. The concepts and viewpoints from learning theory and game theory are used to explain the behaviour of these structures. This work is then extended by presenting a quantitative framework based on Petri Net theory. This formalism provides a powerful means for capturing the information flow in the decision-making process and demonstrating the explicit interactions between decision makers. Additionally, it is also used for the description and analysis of systems that axe characterised as being concurrent, asynchronous, distributed, parallel and/ or stochastic activities. The thesis discusses the limitations of each modelling framework. The thesis proposes an extension to the existing methodologies by presenting a new class of Petri Nets. This extension has resulted in a novel structure which has the additional feature of an embedded stochastic learning automata. An application of this approach to a realistic decision problem demonstrates the impact that the use of an artificial intelligence technique embedded within Petri Nets can have on the performance of decision models

Reinforcement Learning

Brains rule the world, and brain-like computation is increasingly used in computers and electronic devices. Brain-like computation is about processing and interpreting data or directly putting forward and performing actions. Learning is a very important aspect. This book is on reinforcement learning which involves performing actions to achieve a goal. The first 11 chapters of this book describe and extend the scope of reinforcement learning. The remaining 11 chapters show that there is already wide usage in numerous fields. Reinforcement learning can tackle control tasks that are too complex for traditional, hand-designed, non-learning controllers. As learning computers can deal with technical complexities, the tasks of human operators remain to specify goals on increasingly higher levels. This book shows that reinforcement learning is a very dynamic area in terms of theory and applications and it shall stimulate and encourage new research in this field