Fuzzy and tile coding approximation techniques for coevolution in reinforcement learning

PhDThis thesis investigates reinforcement learning algorithms suitable for learning in large state space problems and coevolution. In order to learn in large state spaces, the state space must be collapsed to a computationally feasible size and then generalised about. This thesis presents two new implementations of the classic temporal difference (TD) reinforcement learning algorithm Sarsa that utilise fuzzy logic principles for approximation, FQ Sarsa and Fuzzy Sarsa. The effectiveness of these two fuzzy reinforcement learning algorithms is investigated in the context of an agent marketplace. It presents a practical investigation into the design of fuzzy membership functions and tile coding schemas. A critical analysis of the fuzzy algorithms to a related technique in function approximation, a coarse coding approach called tile coding is given in the context of three different simulation environments; the mountain-car problem, a predator/prey gridworld and an agent marketplace. A further comparison between Fuzzy Sarsa and tile coding in the context of the nonstationary environments of the agent marketplace and predator/prey gridworld is presented. This thesis shows that the Fuzzy Sarsa algorithm achieves a significant reduction of state space over traditional Sarsa, without loss of the finer detail that the FQ Sarsa algorithm experiences. It also shows that Fuzzy Sarsa and gradient descent Sarsa(λ) with tile coding learn similar levels of distinction against a stationary strategy. Finally, this thesis demonstrates that Fuzzy Sarsa performs better in a competitive multiagent domain than the tile coding solution

Equilibrium Analysis of Channel Structure Strategies in Uncertain Environment

Abstract In this paper, we consider a pricing decision problem with two competing supply chains which distribute differentiated but competing products in the same market. Each chain can be vertically integrated or decentralized based on the choice of the manufacturer. The manufacturing costs, sales costs and consumer demands are characterized as uncertain variables, whose distributions are estimated by experienced experts. Meanwhile, uncertainty theory and game theory are employed to formulate the pricing decision problems. The equilibrium behaviors (how the supply chain members make their own pricing decisions on wholesale prices and retailer markups) at operational level under three possible scenarios are derived. Numerical experiments are also given to explore the impacts of the parameters’ uncertain degrees on supply chain members’ pricing decisions. The results demonstrate that the supply chain uncertain factors have great influences on equilibrium prices. In addition, we also evaluate the effects of competing intensity (substitutability) of the two products on the strategy behaviors, vertically integrated channel strategy versus decentralized strategy, of the manufacturers. It is found that the manufacturers are better off to distribute their products through a decentralized channel rather than an integrated one when the substitutability is greater than some value. Besides, the uncertain factors in the supply chain might reduce the value contrast to the one in deterministic case. Some other interesting managerial highlights are also provided in this paper

Robust Fuzzy learning for partially overlapping channels allocation in UAV communication networks

With significantly dynamic characteristics of the new aerial users, the emerging cellular-enabled unmanned aerial vehicle (UAV) communication paradigm raises great challenges to current research of UAV applications. As far as the robust channel allocation is concerned, the high mobility of UAV nodes and the unexpected disturbance of external environment would render most existing methods which rely on definite information and are vulnerable to dynamic environment, become less attractive or even invalid. In this paper, we particularly investigate a cellular-enabled mesh UAV network exploiting partially overlapping channels (POCs), and propose a distributed fuzzy space based learning scheme for POCs allocation to combat the dynamic environment. Rather than the perfect channel state information (CSI) assumption, the dynamic and uncertain CSI of UAVs is characterized by fuzzy number. On this basis, the allocation process can be implemented in a mapped fuzzy space. Integrating fuzzy-logic and game based learning, we formulate the problem of POCs assignment as a fuzzy payoffs game (FPG), and demonstrate the existence of fuzzy Nash equilibrium for our designed FPG. Then, with the derived priority vector in the fuzzy space, the equilibrium solution can be achieved by the proposed algorithm. Numerical simulations demonstrate the advantages of our new scheme

Game theory based multi criteria decision making problem under uncertainty: a case study on Indian Tea Industry

The long-term evolution of multi agent multi criteria decision making (MCDM) and to obtain sustainable decision a novel methodology is proposed based on evolutionary game theory. In this paper multi agent MCDM is represented as an evolutionary game and the evolutionary strategies are defined as sustainable decisions. Here we consider the problem of decision making in Indian Tea Industry. The agents in this game are essentially Indian Tea Estate owner and Indian Tea board. The replicator dynamics of the evolutionary game are studied to obtain evolutionary strategies which could be defined as sustainable strategies. The multi agent MCDM in Indian Tea Industry is considered under different socio-political and Corporate Social Responsibility scenario and groups of Indian Tea Industry. Again, the impacts of imprecision and market volatility on the outcome of some strategies (decisions) are studied. In this paper the imprecision on the impact of the strategies are modelled as fuzzy numbers whereas the market volatility is taken into account as white noise. Hence the MCDM problem for Indian Tea Industry is modelled as a hybrid evolutionary game. The probabilities of strategies are obtained by solving hybrid evolutionary game and could be represented as a Dempster-Shafer belief structure. The simulation results facilitate the Decision Makers to choose the strategies (decisions) under different type of uncertainty

A Temporal Framework for Hypergame Analysis of Cyber Physical Systems in Contested Environments

Game theory is used to model conflicts between one or more players over resources. It offers players a way to reason, allowing rationale for selecting strategies that avoid the worst outcome. Game theory lacks the ability to incorporate advantages one player may have over another player. A meta-game, known as a hypergame, occurs when one player does not know or fully understand all the strategies of a game. Hypergame theory builds upon the utility of game theory by allowing a player to outmaneuver an opponent, thus obtaining a more preferred outcome with higher utility. Recent work in hypergame theory has focused on normal form static games that lack the ability to encode several realistic strategies. One example of this is when a player’s available actions in the future is dependent on his selection in the past. This work presents a temporal framework for hypergame models. This framework is the first application of temporal logic to hypergames and provides a more flexible modeling for domain experts. With this new framework for hypergames, the concepts of trust, distrust, mistrust, and deception are formalized. While past literature references deception in hypergame research, this work is the first to formalize the definition for hypergames. As a demonstration of the new temporal framework for hypergames, it is applied to classical game theoretical examples, as well as a complex supervisory control and data acquisition (SCADA) network temporal hypergame. The SCADA network is an example includes actions that have a temporal dependency, where a choice in the first round affects what decisions can be made in the later round of the game. The demonstration results show that the framework is a realistic and flexible modeling method for a variety of applications

A comprehensive review of hybrid game theory techniques and multi-criteria decision-making methods

More studies trend to hybrid the game theory technique with the multi-criteria decision-making (MCDM) method to aid real-life problems. This paper provides a comprehensive review of the hybrid game theory technique and MCDM method. The fundamentals of game theory concepts and models are explained to make game theory principles clear to the readers. Moreover, the definitions and models are elaborated and classified to the static game, dynamic game, cooperative game and evolutionary game. Therefore, the hybrid game theory technique and MCDM method are reviewed and numerous applications studied from the past works of literature are highlighted. The result of the previous studies shows that the fundamental elements for both frameworks were studied in various ways with most of the past studies tend to integrate the static game with AHP and TOPSIS methods. Also, the integration of game theory techniques and MCDM methods was studied in various applications such as politics, economy, supply chain, engineering, water management problem, allocation problem and telecommunication network selection. The main contribution of the recent studies of employment between game theory technique and MCDM method are analyzed and discussed in detail which includes static and dynamic games in the non-cooperative game, cooperative game, both non-cooperative and cooperative games and evolutionary gam

Integration of Massive Plug-in Hybrid Electric Vehicles into Power Distribution Systems: Modeling, Optimization, and Impact Analysis

With the development of vehicle-to-grid (V2G) technology, it is highly promising to use plug-in hybrid electric vehicles (PHEVs) as a new form of distributed energy resources. However, the uncertainties in the power market and the conflicts among different stakeholders make the integration of PHEVs a highly challenging task. Moreover, the integration of PHEVs may lead to negative effects on the power grid performance if the PHEV fleets are not properly managed. This dissertation studies various aspects of the integration of PHEVs into power distribution systems, including the PHEV load demand modeling, smart charging algorithms, frequency regulation, reliability-differentiated service, charging navigation, and adequacy assessment of power distribution systems. This dissertation presents a comprehensive methodology for modeling the load demand of PHEVs. Based on this stochastic model of PHEV, a two-layer evolution strategy particle swarm optimization (ESPSO) algorithm is proposed to integrate PHEVs into a residential distribution grid. This dissertation also develops an innovative load frequency control system, and proposes a hierarchical game framework for PHEVs to optimize their charging process and participate in frequency regulation simultaneously. The potential of using PHEVs to enable reliability-differentiated service in residential distribution grids has been investigated in this dissertation. Further, an integrated electric vehicle (EV) charging navigation framework has been proposed in this dissertation which takes into consideration the impacts from both the power system and transportation system. Finally, this dissertation proposes a comprehensive framework for adequacy evaluation of power distribution networks with PHEVs penetration. This dissertation provides innovative, viable business models for enabling the integration of massive PHEVs into the power grid. It helps evolve the current power grid into a more reliable and efficient system

Contributions to Game Theory and Management. Vol. III. Collected papers presented on the Third International Conference Game Theory and Management.

The collection contains papers accepted for the Third International Conference Game Theory and Management (June 24-26, 2009, St. Petersburg University, St. Petersburg, Russia). The presented papers belong to the field of game theory and its applications to management. The volume may be recommended for researches and post-graduate students of management, economic and applied mathematics departments.