Learning to Search in Reinforcement Learning

In this thesis, we investigate the use of search based algorithms with deep neural networks to tackle a wide range of problems ranging from board games to video games and beyond. Drawing inspiration from AlphaGo, the first computer program to achieve superhuman performance in the game of Go, we developed a new algorithm AlphaZero. AlphaZero is a general reinforcement learning algorithm that combines deep neural networks with a Monte Carlo Tree search for planning and learning. Starting completely from scratch, without any prior human knowledge beyond the basic rules of the game, AlphaZero managed to achieve superhuman performance in Go, chess and shogi. Subsequently, building upon the success of AlphaZero, we investigated ways to extend our methods to problems in which the rules are not known or cannot be hand-coded. This line of work led to the development of MuZero, a model-based reinforcement learning agent that builds a deterministic internal model of the world and uses it to construct plans in its imagination. We applied our method to Go, chess, shogi and the classic Atari suite of video-games, achieving superhuman performance. MuZero is the first RL algorithm to master a variety of both canonical challenges for high performance planning and visually complex problems using the same principles. Finally, we describe Stochastic MuZero, a general agent that extends the applicability of MuZero to highly stochastic environments. We show that our method achieves superhuman performance in stochastic domains such as backgammon and the classic game of 2048 while matching the performance of MuZero in deterministic ones like Go

Agents for educational games and simulations

This book consists mainly of revised papers that were presented at the Agents for Educational Games and Simulation (AEGS) workshop held on May 2, 2011, as part of the Autonomous Agents and MultiAgent Systems (AAMAS) conference in Taipei, Taiwan. The 12 full papers presented were carefully reviewed and selected from various submissions. The papers are organized topical sections on middleware applications, dialogues and learning, adaption and convergence, and agent applications

Learning Human Behavior From Observation For Gaming Applications

The gaming industry has reached a point where improving graphics has only a small effect on how much a player will enjoy a game. One focus has turned to adding more humanlike characteristics into computer game agents. Machine learning techniques are being used scarcely in games, although they do offer powerful means for creating humanlike behaviors in agents. The first person shooter (FPS), Quake 2, is an open source game that offers a multi-agent environment to create game agents (bots) in. This work attempts to combine neural networks with a modeling paradigm known as context based reasoning (CxBR) to create a contextual game observation (CONGO) system that produces Quake 2 agents that behave as a human player trains them to act. A default level of intelligence is instilled into the bots through contextual scripts to prevent the bot from being trained to be completely useless. The results show that the humanness and entertainment value as compared to a traditional scripted bot have improved, although, CONGO bots usually ranked only slightly above a novice skill level. Overall, CONGO is a technique that offers the gaming community a mode of game play that has promising entertainment value

A Crisis in Physics Education: Games to the Rescue!

An education in Physics develops both strong cognitive and practical skills. These are well-matched to the needs of employers, from engineering to banking. Physics provides the foundation for all engineering and scientific disciplines including computing technologies, aerospace, communication, and also biosciences and medicine. In academe, Physics addresses fundamental questions about the universe, the nature of reality, and of the complex socio-economic systems comprising our daily lives. Yet today, there are emerging concerns about Physics education: Secondary school interest in Physics is falling, as is the number of Physics school teachers. There is clearly a crisis in physics education; recent research has identified principal factors. Starting from a review of these factors, and from recommendations of professional bodies, this paper proposes a novel solution – the use of Computer Games to teach physics to school children, to university undergraduates and to teacher-trainees

Forging Wargamers

How do we establish or improve wargaming education, including sponsors, participants, and future designers? The question stems from the uncomfortable truth that the wargaming discipline has no foundational pipeline, no established pathway from novice to master. Consequently, the wargaming community stands at a dangerous precipice at the convergence of a stagnant labor force and a patchwork system of passing institutional war-gaming knowledge. Unsurprisingly, this can lead to ill-informed sponsors, poorly scoped wargames, an unreliable standard of wargaming expertise, and worst of all, risks the decline of wargaming as an educational and analytical tool. This fundamental challenge is a recurring theme throughout this volume and each author offers their own perspective and series of recommendations

Learning Models in Educational Game Interactions: A Review

Educational games have now been used as innovative media and teaching strategies to achieve more effective learning and have an impact that tends to be very good in the learning process. However, it is important to know and systematically prove that the application of the learning model in the interaction of educational games is indeed feasible to be adopted and has an effect. This paper aims to present empirical evidence of the current situation regarding the application of learning models in the flow of educational game interactions. The method used is a systematic literature review by adopting three main stages, namely: 1) Planning; 2) Implementation; 3) Reporting. Then recommend the ten steps in the systematic literature review process along with the selection process through the test-retest approach. The initial search obtained 1,405,310 papers, then go through the selection stage. The selection process took place at stage B1 with the number of papers that successfully passed 198, at the B2 selection stage there were 102 papers, and we focus 75 papers that have passed a fairly rigorous screening and selection process on the quality assessment process for primary studies, used to answer research objectives and questions. We can confirm and conclude that 75 papers have applied the learning model in educational game interactions. The dominating domain is Education, the type of game that dominates is Educational Game, for the most dominating subjects are Programming, Student Learning Motivation as the most dominating impact, Experimental Design as a trial technique, the most widely used evaluation instruments are Questionnaires and Tests, a population that dominates between 79-2,645 people, and 8 papers to support learning in vocational education

Virtual Reality Games for Motor Rehabilitation

This paper presents a fuzzy logic based method to track user satisfaction without the need for devices to monitor users physiological conditions. User satisfaction is the key to any product’s acceptance; computer applications and video games provide a unique opportunity to provide a tailored environment for each user to better suit their needs. We have implemented a non-adaptive fuzzy logic model of emotion, based on the emotional component of the Fuzzy Logic Adaptive Model of Emotion (FLAME) proposed by El-Nasr, to estimate player emotion in UnrealTournament 2004. In this paper we describe the implementation of this system and present the results of one of several play tests. Our research contradicts the current literature that suggests physiological measurements are needed. We show that it is possible to use a software only method to estimate user emotion

Player agency in interactive narrative: audience, actor & author

The question motivating this review paper is, how can computer-based interactive narrative be used as a constructivist learn- ing activity? The paper proposes that player agency can be used to link interactive narrative to learner agency in constructivist theory, and to classify approaches to interactive narrative. The traditional question driving research in interactive narrative is, ‘how can an in- teractive narrative deal with a high degree of player agency, while maintaining a coherent and well-formed narrative?’ This question derives from an Aristotelian approach to interactive narrative that, as the question shows, is inherently antagonistic to player agency. Within this approach, player agency must be restricted and manip- ulated to maintain the narrative. Two alternative approaches based on Brecht’s Epic Theatre and Boal’s Theatre of the Oppressed are reviewed. If a Boalian approach to interactive narrative is taken the conflict between narrative and player agency dissolves. The question that emerges from this approach is quite different from the traditional question above, and presents a more useful approach to applying in- teractive narrative as a constructivist learning activity

The Morra game as a naturalistic test bed for investigating automatic and voluntary processes in random sequence generation

Morra is a 3,000-years-old hand game of prediction and numbers. The two players reveal their hand simultaneously, presenting a number of fingers between 1 and 5, while calling out a number between 2 and 10. Any player who successfully guesses the summation of fingers revealed by both players scores a point. While the game is extremely fast-paced, making it very difficult for players to achieve a conscious control of their game strategies, expert players regularly outperform non-experts, possibly with strategies residing out of conscious control. In this study, we used Morra as a naturalistic setting to investigate the necessity of attentive control in generation of sequence of items and the ability to proceduralize random number generation, which are both a crucial defensive strategy in Morra and a well-known empirical procedure to test the central executive capacity within the working memory model. We recorded the sequence of numbers generated by expert players in a Morra tournament in Sardinia (Italy) and by undergraduate students enrolled in a course-based research experience (CRE) course at Lawrence Technological University in the United States. Number sequences generated by non-expert and expert players both while playing Morra and in a random number generation task (RNGT) were compared in terms of randomness scores. Results indicate that expert players of Morra largely outperformed non-experts in the randomness scores only within Morra games, whereas in RNGT the two groups were very similar. Importantly, survey data acquired after the games indicate that expert players have very poor conscious recall of their number generation strategies used during the Morra game. Our results indicate that the ability of generating random sequences can be proceduralized and do not necessarily require attentive control. Results are discussed in the framework of the dual processing theory and its automatic-parallel-fast vs.controlled-sequential-slow polarities