NPC AI System Based on Gameplay Recordings

Hästi optimeeritud mitte-mängija tegelased (MMT) on vastaste või meeskonna kaaslastena üheks peamiseks osaks mitme mängija mängudes. Enamus mänguroboteid on ehitatud jäikade süsteemide peal, mis võimaldavad vaid loetud arvu otsuseid ja animatsioone. Kogenud mängijad suudavad eristada mänguroboteid inimmängijatest ning ette ennustada nende liigutusi ja strateegiaid. See alandab mängukogemuse kvaliteeti. Seetõttu, eelistavad mitme mängijaga mängude mängijad mängida pigem inimmängijate kui MMTde vastu. Virtuaalreaalsuse (VR) mängud ja VR mängijad on siiani veel väike osa mängutööstusest ja mitme mängija VR mängud kannatavad mängijabaasi kaotusest, kui mänguomanikud ei suuda leida teisi mängijaid, kellega mängida. See uurimus demonstreerib mängulindistustel põhineva tehisintellekt (TI) süsteemi rakendatavust VR esimese isiku vaates tulistamismängule Vrena. Teemamäng kasutab ebatavalist liikumisesüsteemi, milles mängijad liiguvad otsiankrute abil. VR mängijate liigutuste imiteerimiseks loodi AI süsteem, mis kasutab mängulindistusi navigeerimisandmetena. Süsteem koosneb kolmest peamisest funktsionaalsusest. Need funktsionaalsused on mängutegevuse lindistamine, andmete töötlemine ja navigeerimine. Mängu keskkond on tükeldatud kuubikujulisteks sektoriteks, et vähendada erinevate asukohal põhinevate olekute arvu ning mängutegevus on lindistatud ajaintervallide ja tegevuste põhjal. Loodud mängulogid on segmenteeritud logilõikudeks ning logilõikude abil on loodud otsingutabel. Otsingutabelit kasutatakse MMT agentide navigeerimiseks ning MMTde otsuste langetamise mehanism jäljendab olek-tegevus-tasu kontseptsiooni. Loodud töövahendi kvaliteeti hinnati uuringu põhjal, millest saadi märkimisväärset tagasisidet süsteemi täiustamiseks.A well optimized Non-Player Character (NPC) as an opponent or a teammate is a major part of the multiplayer games. Most of the game bots are built upon a rigid system with numbered decisions and animations. Experienced players can distinguish bots from hu-man players and they can predict bot movements and strategies. This reduces the quality of the gameplay experience. Therefore, multiplayer game players favour playing against human players rather than NPCs. VR game market and VR gamers are still a small frac-tion of the game industry and multiplayer VR games suffer from loss of their player base if the game owners cannot find other players to play with. This study demonstrates the applicability of an Artificial Intelligence (AI) system based on gameplay recordings for a Virtual Reality (VR) First-person Shooter (FPS) game called Vrena. The subject game has an uncommon way of movement, in which the players use grappling hooks to navigate. To imitate VR players’ movements and gestures an AI system is developed which uses gameplay recordings as navigation data. The system contains three major functionality. These functionalities are gameplay recording, data refinement, and navigation. The game environment is sliced into cubic sectors to reduce the number of positional states and gameplay is recorded by time intervals and actions. Produced game logs are segmented into log sections and these log sections are used for creating a look-up table. The lookup table is used for navigating the NPC agent and the decision mechanism followed a way similar to the state-action-reward concept. The success of the developed tool is tested via a survey, which provided substantial feedback for improving the system

Mining Experiential Patterns from Game-Logs of Board Game

In board games, game-logs record past game processes, which can be regarded as an accumulation of experience. Similar to a real person, a computer player can gradually increase its skill by learning from game-logs. Therefore, the game becomes more interesting. This paper proposes an extensible approach to mine experiential patterns from increasing game-logs. The computer player improves its strategies by utilizing these growing patterns, just as it acquires experience. To evaluate the effect and performance of the approach, we designed a sample board game as a test platform and elaborated an experiment consisting of a series of tests. Experimental results show that our approach is effective and efficient

Mining Experiential Patterns from Game-Logs of Board Game

In board games, game-logs record past game processes, which can be regarded as an accumulation of experience. Similar to a real person, a computer player can gradually increase its skill by learning from game-logs. Therefore, the game becomes more interesting. This paper proposes an extensible approach to mine experiential patterns from increasing game-logs. The computer player improves its strategies by utilizing these growing patterns, just as it acquires experience. To evaluate the effect and performance of the approach, we designed a sample board game as a test platform and elaborated an experiment consisting of a series of tests. Experimental results show that our approach is effective and efficient

Uma nova abordagem de aprendizagem de máquina combinando elicitação automática de casos, aprendizagem por reforço e mineração de padrões sequenciais para agentes jogadores de damas

ake into account, in addition to the environment, the minimizing action of an opponent (such as in games), it is fundamental that the agent has the ability to progressively trace a proĄle of its adversary that aids it in the process of selecting appropriate actions. However, it would be unsuitable to construct an agent with a decision-making system based on only the elaboration of this proĄle, as this would prevent the agent from having its Şown identityŤ, which would leave it at the mercy of its opponent. Following this direction, this work proposes an automatic hybrid Checkers player, called ACE-RL-Checkers, equipped with a dynamic decision-making mechanism, which adapts to the proĄle of its opponent over the course of the game. In such a system, the action selection process (moves) is conducted through a composition of Multi-Layer Perceptron Neural Network and case library. In the case, Neural Network represents the ŞidentityŤ of the agent, i.e., it is an already trained static decision-making module and makes use of the Reinforcement Learning TD( ) techniques. On the other hand, the case library represents the dynamic decision-making module of the agent, which is generated by the Automatic Case Elicitation technique (a particular type of Case-Based Reasoning). This technique has a pseudo-random exploratory behavior, which makes the dynamic decision-making on the part of the agent to be directed, either by the game proĄle of the opponent or randomly. However, when devising such an architecture, it is necessary to avoid the following problem: due to the inherent characteristics of the Automatic Case Elicitation technique, in the game initial phases, in which the quantity of available cases in the library is extremely low due to low knowledge content concerning the proĄle of the adversary, the decisionmaking frequency for random decisions is extremely high, which would be detrimental to the performance of the agent. In order to attack this problem, this work also proposes to incorporate onto the ACE-RL-Checkers architecture a third module composed of a base of experience rules, extracted from games played by human experts, using a Sequential Pattern Mining technique. The objective behind using such a base is to reĄne and accelerate the adaptation of the agent to the proĄle of its opponent in the initial phases of their confrontations. Experimental results conducted in tournaments involving ACE-RL-Checkers and other agents correlated with this work, conĄrm the superiority of the dynamic architecture proposed herein.Fundação de Amparo a Pesquisa do Estado de Minas GeraisTese (Doutorado)Agentes que operam em ambientes onde as tomadas de decisão precisam levar em conta, além do ambiente, a atuação minimizadora de um oponente (tal como nos jogos), é fundamental que o agente seja dotado da habilidade de, progressivamente, traçar um perĄl de seu adversário que o auxilie em seu processo de seleção de ações apropriadas. Entretanto, seria improdutivo construir um agente com um sistema de tomada de decisão baseado apenas na elaboração desse perĄl, pois isso impediria o agente de ter uma Şidentidade própriaŤ, o que o deixaria a mercê de seu adversário. Nesta direção, este trabalho propõe um sistema automático jogador de Damas híbrido, chamado ACE-RL-Checkers, dotado de um mecanismo dinâmico de tomada de decisões que se adapta ao perĄl de seu oponente no decorrer de um jogo. Em tal sistema, o processo de seleção de ações (movimentos) é conduzido por uma composição de Rede Neural de Perceptron Multicamadas e biblioteca de casos. No caso, a Rede Neural representa a ŞidentidadeŤ do agente, ou seja, é um módulo tomador de decisões estático já treinado e que faz uso da técnica de Aprendizagem por Reforço TD( ). Por outro lado, a biblioteca de casos representa o módulo tomador de decisões dinâmico do agente que é gerada pela técnica de Elicitação Automática de Casos (um tipo particular de Raciocínio Baseado em Casos). Essa técnica possui um comportamento exploratório pseudo-aleatório que faz com que a tomada de decisão dinâmica do agente seja guiada, ora pelo perĄl de jogo do adversário, ora aleatoriamente. Contudo, ao conceber tal arquitetura, é necessário evitar o seguinte problema: devido às características inerentes à técnica de Elicitação Automática de Casos, nas fases iniciais do jogo Ű em que a quantidade de casos disponíveis na biblioteca é extremamente baixa em função do exíguo conhecimento do perĄl do adversário Ű a frequência de tomadas de decisão aleatórias seria muito elevada, o que comprometeria o desempenho do agente. Para atacar tal problema, este trabalho também propõe incorporar à arquitetura do ACE-RLCheckers um terceiro módulo, composto por uma base de regras de experiência extraída a partir de jogos de especialistas humanos, utilizando uma técnica de Mineração de Padrões Sequenciais. O objetivo de utilizar tal base é reĄnar e acelerar a adaptação do agente ao perĄl de seu adversário nas fases iniciais dos confrontos entre eles. Resultados experimentais conduzidos em torneio envolvendo ACE-RL-Checkers e outros agentes correlacionados com este trabalho, conĄrmam a superioridade da arquitetura dinâmica aqui proposta

Aprimorando o processo de aprendizagem e alocação de agentes inteligentes em plataformas multiagentes: aplicação no domínio do jogo de damas

One of the fundamental requirements for a unsupervised multiagent system to reach its objectives is that the agents that make up the system possess specific and complementary abilities, which allow them to act as specialists in the environments where they were trained. The adequate representation of these environments is fundamental to both the learning and to the good performance on the part of the agents, mainly when these act in competitive environments that possess an elevated state space. Likewise, the decisions from multiagent systems, through their allocation of adequate agents into particular situations that occur in these environments, are crucial in order that these successfully reach their objectives. In this sense, the present work presents three new approaches to optimize the performance of multiagent systems, which improves: the architecture and the learning process of the agents that make up the multiagent system; the representation of relevant information of the environments where these agents perform, as well as the process of allocating the adequate agent for performing in distinct situations that occur in these environments. Due to the spatial and technical complexity, the game of Checkers was used as the developmental and evaluative environment for these approaches, which were implemented onto the automatic player MP-Draughts. This player corresponds to a unsupervised multiagent system composed of specialist player agents in distinct phases of a game. In order to implement the proposed approaches onto the MP-Draughts architecture, the following work sequence was adopted: initially, an adaptive neural network was developed, ASONDE, which was used in the MP-Draughts architecture to define the knowledge profiles (clusters) necessary for representing the endgame phase, on which the specialist agents should be trained. Following on, an automatic features selection approach based on the frequent pattern mining was implemented, which extracts the most adequate features to represent the different environments (boards) that can occur during the performance of the multi-agent. Finally, a method for the allocation of agents was developed, which combined clustering artificial neural networks and exception rules, which together are responsible for indicating the most suitable agents to act in the different situations of a game. The partial results obtained from the implementation of each approach, as well as the final result, which applies all these into the MP-Draughts architecture, confirm that these were efficient in dealing with the problems for which they were proposed, in addition to contributing to the general performance of the multi-agent system.FAPEMIG - Fundação de Amparo a Pesquisa do Estado de Minas GeraisTese (Doutorado)Um dos requisitos fundamentais para que um sistema multiagente não supervisionado atinja seus objetivos é que os agentes que o compõem possuam habilidades específicas e complementares que lhe permitam atuar como especialistas nos ambientes em que foram treinados. A representação adequada desses ambientes é fundamental para o aprendizado e para a boa performance dos agentes, principalmente quando esses atuam em ambientes competitivos que possuem elevado espaço de estados. Do mesmo modo, as decisões do sistema multiagente em alocar os agentes adequados para atuarem em determinadas situações que ocorrem nesses ambientes são cruciais para que este atinja, com êxito, seus objetivos. Nesse sentido, o presente trabalho apresenta três novas abordagens para otimizar o desempenho de sistemas multiagentes, as quais aprimoram: a arquitetura e o processo de aprendizagem dos agentes que compõem o sistema multiagente, a representação das informações relevantes dos ambientes de atuação desses agentes, assim como o processo de alocação dos agentes adequados para atuarem nas distintas situações que ocorrem nesses ambientes. Devido à sua complexidade espacial e técnica, o jogo Damas foi utilizado como ambiente de desenvolvimento e avaliação dessas abordagens, as quais foram implementadas na arquitetura do jogador automático MP-Draughts. Tal jogador corresponde a um sistema multiagente não supervisionado composto por agentes jogadores especialistas em fases distintas de um jogo. Para a implementação das abordagens propostas na arquitetura do MP-Draughts, foi adotada a seguinte sequência de trabalho: inicialmente, foi desenvolvida uma rede neural adaptativa, a ASONDE, que foi utilizada na arquitetura do MP-Draughts para definir os perfis (clusters) de conhecimentos necessários para representar a fase de final de jogo, nos quais os agentes especialistas devem ser treinados. Na sequência, foi implementada uma abordagem de seleção automática de características baseada na mineração de padrões frequentes, a qual extrai as mais adequadas para representar os diferentes ambientes (tabuleiros) que podem ocorrer durante a atuação do multiagente. Finalmente, foi desenvolvido um método de alocação de agentes que combina redes neurais artificiais e regras de exceção, as quais em conjunto, são responsáveis por indicar os agentes mais adequados para atuarem nas distintas situações de um jogo. Os resultados parciais obtidos da implementação de cada abordagem, assim como o resultado final que aplica todas elas na arquitetura no MP-Draughts, confirmam que as mesmas foram eficientes para tratar os problemas para os quais foram propostas, além de contribuírem para o desempenho geral do sistema multiagente

A gauge-invariant, symmetry-preserving truncation of JIMWLK

The colour glass condensate captures quantum chromodynamics in its application to high-energy collider experiments in the spirit of an effective field theory. In deeply inelastic lepton-hadron scattering experiments, as well as in hadron-hadron collisions, the internal degrees of freedom of in-state hadrons are dominated by a dense medium of gluonic matter called the colour glass condensate. Interactions with this medium by some (dilute) probe are most naturally described in terms of Wilson-lines and their correlators. The energy-dependence of these correlators is given by the JIMWLK (Jalilian-Marian+Iancu+McLerran+Weigert+Leonidov+Kovner) equa- tion which, when applied to a correlator, generates an infinite tower of coupled Dyson-Schwinger- like equations referred to as a Balitsky Hierarchy. In this thesis, I present a novel method for truncating, in a gauge-invariant and symmetry- preserving manner, the Balitsky hierarchy associated with matrices of Wilson-line correlators. This truncation is realized by parameterizing the energy-dependence of the symmetric and anti- symmetric parts of these matrices independently via energy-evolution operators which evolve ini- tial conditions in a manner akin to the time-evolution of Hermitian operators in the Heisenberg picture of quantum mechanics. These energy-evolution operators are path-ordered exponentials whose exponents are expanded in terms of energy-dependent "colour structure functions". I show how the properties of contributions to the expansion of these exponents (at each order in the expansion) are constrained by the group theory of SU(Nc)