Multiobjective Monte Carlo Tree Search for Real-Time Games

Multiobjective optimization has been traditionally a matter of study in domains like engineering or finance, with little impact on games research. However, action-decision based on multiobjective evaluation may be beneficial in order to obtain a high quality level of play. This paper presents a multiobjective Monte Carlo tree search algorithm for planning and control in real-time game domains, those where the time budget to decide the next move to make is close to 40 ms. A comparison is made between the proposed algorithm, a single-objective version of Monte Carlo tree search and a rolling horizon implementation of nondominated sorting evolutionary algorithm II (NSGA-II). Two different benchmarks are employed, deep sea treasure (DST) and the multiobjective physical traveling salesman problem (MO-PTSP). Using the same heuristics on each game, the analysis is focused on how well the algorithms explore the search space. Results show that the algorithm proposed outperforms NSGA-II. Additionally, it is also shown that the algorithm is able to converge to different optimal solutions or the optimal Pareto front (if achieved during search)

A spatially-structured PCG method for content diversity in a Physics-based simulation game

This paper presents a spatially-structured evolutionary algorithm (EA) to procedurally generate game maps of di ferent levels of di ficulty to be solved, in Gravityvolve!, a physics-based simulation videogame that we have implemented and which is inspired by the n- body problem, a classical problem in the fi eld of physics and mathematics. The proposal consists of a steady-state EA whose population is partitioned into three groups according to the di ficulty of the generated content (hard, medium or easy) which can be easily adapted to handle the automatic creation of content of diverse nature in other games. In addition, we present three fitness functions, based on multiple criteria (i.e:, intersections, gravitational acceleration and simulations), that were used experimentally to conduct the search process for creating a database of maps with di ferent di ficulty in Gravityvolve!.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech

Multi-objective tree search approaches for general video game playing

The design of algorithms for Game AI agents usually focuses on the single objective of winning, or maximizing a given score. Even if the heuristic that guides the search (for reinforcement learning or evolutionary approaches) is composed of several factors, these typically provide a single numeric value (reward or fitness, respectively) to be optimized. Multi-Objective approaches are an alternative concept to face these problems, as they try to optimize several objectives, often contradictory, at the same time. This paper proposes for the first time a study of Multi-Objective approaches for General Video Game playing, where the game to be played is not known a priori by the agent. The experimental study described here compares several algorithms in this setting, and the results suggest that Multi-Objective approaches can perform even better than their single-objective counterparts

Improving Performance Insensitivity of Large-scale Multiobjective Optimization via Monte Carlo Tree Search

The large-scale multiobjective optimization problem (LSMOP) is characterized by simultaneously optimizing multiple conflicting objectives and involving hundreds of decision variables. {Many real-world applications in engineering fields can be modeled as LSMOPs; simultaneously, engineering applications require insensitivity in performance.} This requirement usually means that the results from the algorithm runs should not only be good for every run in terms of performance but also that the performance of multiple runs should not fluctuate too much, i.e., the algorithm shows good insensitivity. Considering that substantial computational resources are requested for each run, it is essential to improve upon the performance of the large-scale multiobjective optimization algorithm, as well as the insensitivity of the algorithm. However, existing large-scale multiobjective optimization algorithms solely focus on improving the performance of the algorithms, leaving the insensitivity characteristics unattended. {In this work, we propose an evolutionary algorithm for solving LSMOPs based on Monte Carlo tree search, the so-called LMMOCTS, which aims to improve the performance and insensitivity for large-scale multiobjective optimization problems.} The proposed method samples the decision variables to construct new nodes on the Monte Carlo tree for optimization and evaluation. {It selects nodes with good evaluation for further search to reduce the performance sensitivity caused by large-scale decision variables.} We compare the proposed algorithm with several state-of-the-art designs on different benchmark functions. We also propose two metrics to measure the sensitivity of the algorithm. The experimental results confirm the effectiveness and performance insensitivity of the proposed design for solving large-scale multiobjective optimization problems.Comment: 12 pages, 11 figure

Hybrid Monte Carlo tree search based multi-objective scheduling

As markets demand targeted products for highly differentiated use cases, the number of variants in production increases, whilst the volume per variant decreases. Different product variants result in differences in work content on workstation level which cause takt time losses and result in a poor utilization. In this context, matrix-structured production systems with neither temporal nor spacial linkage emerged to reduce the effects of different work content on the entire production system. However, matrix-structured production systems require far more complex production control. To that end, this paper presents a scheduling approach. The proposed scheduling system considers variable process sequences and their allocation to different workstations in order to optimize scheduling objectives. This contribution presents a Monte Carlo tree search based optimizer combined with local search as post optimizer to derive schedules in a short time span to enabling reactive scheduling. The application of the scheduler to a benchmark problem and an industrial scheduling problem demonstrates the quality of the results and illustrates how the scheduler reassigns the work content dynamically

Online evolution for multi-action adversarial games

We present Online Evolution, a novel method for playing turn-based multi-action adversarial games. Such games, which include most strategy games, have extremely high branching factors due to each turn having multiple actions. In Online Evolution, an evolutionary algorithm is used to evolve the combination of atomic actions that make up a single move, with a state evaluation function used for fitness. We implement Online Evolution for the turn-based multi-action game Hero Academy and compare it with a standard Monte Carlo Tree Search implementation as well as two types of greedy algorithms. Online Evolution is shown to outperform these methods by a large margin. This shows that evolutionary planning on the level of a single move can be very effective for this sort of problems

A practical guide to multi-objective reinforcement learning and planning

Real-world sequential decision-making tasks are generally complex, requiring trade-offs between multiple, often conflicting, objectives. Despite this, the majority of research in reinforcement learning and decision-theoretic planning either assumes only a single objective, or that multiple objectives can be adequately handled via a simple linear combination. Such approaches may oversimplify the underlying problem and hence produce suboptimal results. This paper serves as a guide to the application of multi-objective methods to difficult problems, and is aimed at researchers who are already familiar with single-objective reinforcement learning and planning methods who wish to adopt a multi-objective perspective on their research, as well as practitioners who encounter multi-objective decision problems in practice. It identifies the factors that may influence the nature of the desired solution, and illustrates by example how these influence the design of multi-objective decision-making systems for complex problems. © 2022, The Author(s)

Dynamic multi-objective optimisation using deep reinforcement learning::benchmark, algorithm and an application to identify vulnerable zones based on water quality

Dynamic multi-objective optimisation problem (DMOP) has brought a great challenge to the reinforcement learning (RL) research area due to its dynamic nature such as objective functions, constraints and problem parameters that may change over time. This study aims to identify the lacking in the existing benchmarks for multi-objective optimisation for the dynamic environment in the RL settings. Hence, a dynamic multi-objective testbed has been created which is a modified version of the conventional deep-sea treasure (DST) hunt testbed. This modified testbed fulfils the changing aspects of the dynamic environment in terms of the characteristics where the changes occur based on time. To the authors’ knowledge, this is the first dynamic multi-objective testbed for RL research, especially for deep reinforcement learning. In addition to that, a generic algorithm is proposed to solve the multi-objective optimisation problem in a dynamic constrained environment that maintains equilibrium by mapping different objectives simultaneously to provide the most compromised solution that closed to the true Pareto front (PF). As a proof of concept, the developed algorithm has been implemented to build an expert system for a real-world scenario using Markov decision process to identify the vulnerable zones based on water quality resilience in São Paulo, Brazil. The outcome of the implementation reveals that the proposed parity-Q deep Q network (PQDQN) algorithm is an efficient way to optimise the decision in a dynamic environment. Moreover, the result shows PQDQN algorithm performs better compared to the other state-of-the-art solutions both in the simulated and the real-world scenario

A Practical Guide to Multi-Objective Reinforcement Learning and Planning

Real-world decision-making tasks are generally complex, requiring trade-offs between multiple, often conflicting, objectives. Despite this, the majority of research in reinforcement learning and decision-theoretic planning either assumes only a single objective, or that multiple objectives can be adequately handled via a simple linear combination. Such approaches may oversimplify the underlying problem and hence produce suboptimal results. This paper serves as a guide to the application of multi-objective methods to difficult problems, and is aimed at researchers who are already familiar with single-objective reinforcement learning and planning methods who wish to adopt a multi-objective perspective on their research, as well as practitioners who encounter multi-objective decision problems in practice. It identifies the factors that may influence the nature of the desired solution, and illustrates by example how these influence the design of multi-objective decision-making systems for complex problems