Fuzzy and tile coding approximation techniques for coevolution in reinforcement learning

PhDThis thesis investigates reinforcement learning algorithms suitable for learning in large state space problems and coevolution. In order to learn in large state spaces, the state space must be collapsed to a computationally feasible size and then generalised about. This thesis presents two new implementations of the classic temporal difference (TD) reinforcement learning algorithm Sarsa that utilise fuzzy logic principles for approximation, FQ Sarsa and Fuzzy Sarsa. The effectiveness of these two fuzzy reinforcement learning algorithms is investigated in the context of an agent marketplace. It presents a practical investigation into the design of fuzzy membership functions and tile coding schemas. A critical analysis of the fuzzy algorithms to a related technique in function approximation, a coarse coding approach called tile coding is given in the context of three different simulation environments; the mountain-car problem, a predator/prey gridworld and an agent marketplace. A further comparison between Fuzzy Sarsa and tile coding in the context of the nonstationary environments of the agent marketplace and predator/prey gridworld is presented. This thesis shows that the Fuzzy Sarsa algorithm achieves a significant reduction of state space over traditional Sarsa, without loss of the finer detail that the FQ Sarsa algorithm experiences. It also shows that Fuzzy Sarsa and gradient descent Sarsa(λ) with tile coding learn similar levels of distinction against a stationary strategy. Finally, this thesis demonstrates that Fuzzy Sarsa performs better in a competitive multiagent domain than the tile coding solution

On Video Game Balancing: Joining Player- and Data-Driven Analytics

Balancing is, especially among players, a highly debated topic of video games. Whether a game is sufficiently balanced greatly influences its reception, player satisfaction, churn rates and success. Yet, conceptions about the definition of balance diverge across industry, academia and players, and different understandings of designing balance can lead to worse player experiences than actual imbalances. This work accumulates concepts of balancing video games from industry and academia and introduces a player-driven approach to optimize player experience and satisfaction. Using survey data from 680 participants and empirically recorded data of over 4 million in-game fights of Guild Wars 2, we aggregate player opinions and requirements, contrast them to the status quo and approach a democratized quantitative technique to approximate closer configurations of balance. We contribute a strategy of refining balancing notions, a methodology of tailoring balance to the actual player base and point to an exemplary artifact that realizes this process.Comment: 25 pages, 5 figure

A meta-architecture analysis for a coevolved system-of-systems

Modern engineered systems are becoming increasingly complex. This is driven in part by an increase in the use of systems-of-systems and network-centric concepts to improve system performance. The growth of systems-of-systems allows stakeholders to achieve improved performance, but also presents new challenges due to increased complexity. These challenges include managing the integration of asynchronously developed systems and assessing SoS performance in uncertain environments. Many modern systems-of-systems must adapt to operating environment changes to maintain or improve performance. Coevolution is the result of the system and the environment adapting to changes in each other to obtain a performance advantage. The complexity that engineered systems-of-systems exhibit poses challenges to traditional systems engineering approaches. Systems engineers are presented with the problem of understanding how these systems can be designed or adapted given these challenges. Understanding how the environment influences system-of-systems performance allows systems engineers to target the right set of capabilities when adapting the system for improved performance. This research explores coevolution in a counter-trafficking system-of-systems and develops an approach to demonstrate its impacts. The approach implements a trade study using swing weights to demonstrate the influence of coevolution on stakeholder value, develops a novel future architecture to address degraded capabilities, and demonstrates the impact of the environment on system performance using simulation. The results provide systems engineers with a way to assess the impacts of coevolution on the system-of-systems, identify those capabilities most affected, and explore alternative meta-architectures to improve system-of-systems performance in new environments --Abstract, page iii

An evolutionary approach to military history

This paper provides a new way of analysing the concept of change within the field of military history. The proposal is based on the use of complex adaptive systems and evolutionary theory. We introduce the concepts of selection, adaptation and coevolution to explain how war is managed in different societies, and game theory to explore decision-making processes of commanders. We emphasize the value of integrating formal modeling and computational simulations in order to apply the approach to real case studies. Our conclusions outline the advantages of an evolutionary military history in the difficult task of understanding the causes of transformation in past battlefields and armies.Este artículo explora una nueva forma de analizar el concepto de cambio en el campo de la historia militar. La propuesta se basa en el uso de sistemas complejos adaptativos y teoría evolutiva. Introducimos los conceptos de selección, adaptación y co-evolución para explicar cómo las diferentes sociedades humanas gestionan los conflictos bélicos, y la teoría de juegos para explorar los procesos de toma de decisiones de los comandantes. Se enfatiza el valor de integrar modelos formales y simulación computacionals a la hora de aplicar esta aproximación a casos de estudio reales. Las conclusiones resumen las ventajas de una historia militar evolutiva en la difícil tarea de explicar las transformaciones de ejércitos y conflictos pasados

Comparing dynamitic difficulty adjustment and improvement in action game

A thesis submitted to the University of Bedfordshire in partial fulfilment of the requirements for the degree of Master ResearchDesigning a game difficulty is one of the key things as a game designer. Player will be feeling boring when the game designer makes the game too easy or too hard. In the past decades, most of single player games can allow players to choose the game difficulty either easy, normal or hard which define the overall game difficulty. In action game, these options are lack of flexibility and they are unsuitable to the player skill to meet the game difficulty. By using Dynamic Difficulty Adjustment (DDA), it can change the game difficulty in real time and it can match different player skills. In this paper, the final goal is the comparison of the three DDA systems in action game and apply an improved DDA. In order to apply a new improved DDA, this thesis will evaluate three chosen DDA systems with chosen action decision based AI for action game. A new DDA measurement formula is applied to the comparing section

Turing Learning: Advances and Applications

Turing Learning is the family of algorithms where models and discriminators are generated in a competitive setting. This thesis concerns the coevolutionary framework of Turing Learning and investigates the advances for improved model accuracy and the applications in robotic systems. Advances proposed in this thesis are as follows: an interactive approach to enable the discriminator to genuinely influence the data sampling process; a hybrid formulation to combine the benefits of the interactive discriminator in improving model accuracy and the advantages of the passive discriminator for reducing training cost; an exclusiveness reward mechanism to promote candidates with the exclusive performance during the coevolutionary process. Applications presented in this thesis are as follows: an approach for a mobile robotic agent to automatically infer its sensor configuration; an approach for the robot agent to automatically calibrate its sensor reading; a novel approach to infer swarm behaviours from their effects on the environment. The interactive approach has been validated in the inference of sensor configuration and calibration model, leading to the self-modelling/self-discovery process of robotic agents. Results suggest an improved model accuracy with the interactive approach in both cases, compared with the passive approach. The hybrid formulation and the exclusiveness reward mechanism have been demonstrated in the inference of the calibration model. Results show that almost half of the training cost can be reduced without a decrease in model accuracy by applying the hybrid formulation. The novel reward mechanism can accelerate the convergence without a decrease in model accuracy. The indirect way of inferring swarm behaviours requires a small amount of training and reveals novel behavioural controllers for individual robots

Essays on Business Value Creation in Digital Platform Ecosystems

Digital platforms and the surrounding ecosystems have garnered great interest from researchers and practitioners. Notwithstanding this attention, it remains unclear how and when digital platforms create business value for platform owners and complementors. This three-essay dissertation focuses on understanding business value creation in digital platform ecosystems. The first essay reviews and synthesizes literature across disciplines and offers an integrative framework of digital platform business value. Advised by the findings from the review, the second and third essays focus on the value creation for platform complementors. The second essay examines how IT startups entering a platform ecosystem at different times can strategically design their products (i.e., product diversification across platform architectural layers and product differentiation) to gain competitive advantages. Longitudinal evidence from the Hadoop ecosystem demonstrates that product diversification has an inverted U-shaped relationship with complementors success, and such an effect is more salient for earlier entrants than later entrants. Earlier entrants should develop products that are similar to other ecosystem competitors to reduce uncertainty whereas later entrants are advised to explore market niche and differentiate their products.The third essay investigates how platform complementors strategies and products co-evolve over time in the co-created ecosystem network environment. Our longitudinal analysis of the Hadoop ecosystem indicates that complementors technological architecture coverage and alliance exploration strategies increase their product evolution rate. In turn, complementors with faster product evolution are more likely to explore new partners but less likely to cover a wider range of the focal platforms technological layers in subsequent periods. Network density, co-created by all platform complementors, weakens the effects of complementors strategies on their product evolution but amplifies the effects of past product evolutions on strategies.This three-essay dissertation uncovers various understudied competitive strategies in the digital platform context and enriches our understanding of business value creation in digital platform ecosystems

TIME BALANCING OF COMPUTER GAMES USING ADAPTIVE TIME-VARIANT MINIGAMES

Game designers spend a great deal of time developing balanced game experiences. However, differences in player ability, hardware capacity (e.g. network connections) or real-world elements (as in mixed-reality games), make it difficult to balance games for different players in different conditions. In this research, adaptive time-variant minigames have been introduced as a method of addressing the challenges in time balancing as a part of balancing players of games. These minigames were parameterized to allow both a guaranteed minimum play time (the minimum time to complete a minigames to address the fixed temporal constraints) and dynamic adaptability (the ability of adapting the game during the game play to address temporal variations caused by individual differences). Three time adaptation algorithms have been introduced in this research and the interaction between adaptive algorithm, game mechanic, and game difficulty were analyzed in controlled experiments. The studies showed that there are significant effects and interactions for all three factors, confirming the initial hypothesis that these processes were important and linked to each other. Furthermore, the studies revealed that finer temporal granularity leads to less-perceptible adaptation and smaller deviations in game completion times. The results also provided evidence that adaptation mechanisms allow accurate prediction of play time. The designed minigames were valuable in helping to balance temporal asymmetries in a real mixed-reality game. It was also found that these adaptation algorithms did not interrupt the overall play experience