Modelling motivation for experience-based attention focus in reinforcement learning

Computational models of motivation are software reasoning processes designed to direct, activate or organise the behaviour of artificial agents. Models of motivation inspired by psychological motivation theories permit the design of agents with a key reasoning characteristic of natural systems: experience-based attention focus. The ability to focus attention is critical for agent behaviour in complex or dynamic environments where only small amounts of available information is relevant at a particular time. Furthermore, experience-based attention focus enables adaptive behaviour that focuses on different tasks at different times in response to an agent’s experiences in its environment. This thesis is concerned with the synthesis of motivation and reinforcement learning in artificial agents. This extends reinforcement learning to adaptive, multi-task learning in complex, dynamic environments. Reinforcement learning algorithms are computational approaches to learning characterised by the use of reward or punishment to direct learning. The focus of much existing reinforcement learning research has been on the design of the learning component. In contrast, the focus of this thesis is on the design of computational models of motivation as approaches to the reinforcement component that generates reward or punishment. The primary aim of this thesis is to develop computational models of motivation that extend reinforcement learning with three key aspects of attention focus: rhythmic behavioural cycles, adaptive behaviour and multi-task learning in complex, dynamic environments. This is achieved by representing such environments using context-free grammars, modelling maintenance tasks as observations of these environments and modelling achievement tasks as events in these environments. Motivation is modelled by processes for task selection, the computation of experience-based reward signals for different tasks and arbitration between reward signals to produce a motivation signal. Two specific models of motivation based on the experience-oriented psychological concepts of interest and competence are designed within this framework. The first models motivation as a function of environmental experiences while the second models motivation as an introspective process. This thesis synthesises motivation and reinforcement learning as motivated reinforcement learning agents. Three models of motivated reinforcement learning are presented to explore the combination of motivation with three existing reinforcement learning components. The first model combines motivation with flat reinforcement learning for highly adaptive learning of behaviours for performing multiple tasks. The second model facilitates the recall of learned behaviours by combining motivation with multi-option reinforcement learning. In the third model, motivation is combined with an hierarchical reinforcement learning component to allow both the recall of learned behaviours and the reuse of these behaviours as abstract actions for future learning. Because motivated reinforcement learning agents have capabilities beyond those of existing reinforcement learning approaches, new techniques are required to measure their performance. The secondary aim of this thesis is to develop metrics for measuring the performance of different computational models of motivation with respect to the adaptive, multi-task learning they motivate. This is achieved by analysing the behaviour of motivated reinforcement learning agents incorporating different motivation functions with different learning components. Two new metrics are introduced that evaluate the behaviour learned by motivated reinforcement learning agents in terms of the variety of tasks learned and the complexity of those tasks. Persistent, multi-player computer game worlds are used as the primary example of complex, dynamic environments in this thesis. Motivated reinforcement learning agents are applied to control the non-player characters in games. Simulated game environments are used for evaluating and comparing motivated reinforcement learning agents using different motivation and learning components. The performance and scalability of these agents are analysed in a series of empirical studies in dynamic environments and environments of progressively increasing complexity. Game environments simulating two types of complexity increase are studied: environments with increasing numbers of potential learning tasks and environments with learning tasks that require behavioural cycles comprising more actions. A number of key conclusions can be drawn from the empirical studies, concerning both different computational models of motivation and their combination with different reinforcement learning components. Experimental results confirm that rhythmic behavioural cycles, adaptive behaviour and multi-task learning can be achieved using computational models of motivation as an experience-based reward signal for reinforcement learning. In dynamic environments, motivated reinforcement learning agents incorporating introspective competence motivation adapt more rapidly to change than agents motivated by interest alone. Agents incorporating competence motivation also scale to environments of greater complexity than agents motivated by interest alone. Motivated reinforcement learning agents combining motivation with flat reinforcement learning are the most adaptive in dynamic environments and exhibit scalable behavioural variety and complexity as the number of potential learning tasks is increased. However, when tasks require behavioural cycles comprising more actions, motivated reinforcement learning agents using a multi-option learning component exhibit greater scalability. Motivated multi-option reinforcement learning also provides a more scalable approach to recall than motivated hierarchical reinforcement learning. In summary, this thesis makes contributions in two key areas. Computational models of motivation and motivated reinforcement learning extend reinforcement learning to adaptive, multi-task learning in complex, dynamic environments. Motivated reinforcement learning agents allow the design of non-player characters for computer games that can progressively adapt their behaviour in response to changes in their environment

NPCs as People, Too: The Extreme AI Personality Engine

PK Dick once asked “Do Androids Dream of Electric Sheep?” In video games, a similar question could be asked of non-player characters: Do NPCs have dreams? Can they live and change as humans do? Can NPCs have personalities, and can these develop through interactions with players, other NPCs, and the world around them? Despite advances in personality AI for games, most NPCs are still undeveloped and undeveloping, reacting with flat affect and predictable routines that make them far less than human— in fact, they become little more than bits of the scenery that give out parcels of information. This need not be the case. Extreme AI, a psychology-based personality engine, creates adaptive NPC personalities. Originally developed as part of the thesis “NPCs as People: Using Databases and Behaviour Trees to Give Non-Player Characters Personality,” Extreme AI is now a fully functioning personality engine using all thirty facets of the Five Factor model of personality and an AI system that is live throughout gameplay. This paper discusses the research leading to Extreme AI; develops the ideas found in that thesis; discusses the development of other personality engines; and provides examples of Extreme AI’s use in two game demos

City of dred – a tabletop RPG learning experience

Learning experiences are not typically used to describe formal learning activities, such as in classroom, transmissive methods. Centred in the student, this term describe that the learner is experiencing something that, hopefully, contributes to a change in thinking, understanding, or behaviour afterwards. For this to happen, learning experiences should be active, meaningful, with social meaning, integrative, and diversified. We consider active learning experiences when the student has the main learning role. They should provide knowledge and skills that directly contribute to the learner’s ability to perform more effectively in the context of workplace learning. Sharing and cooperation is fundamental, allowing the learner to interact with other active learners. The inherent increase in complexity demands the integration of different dimensions of knowledge, better achieved through diversified strategies. In this context, teaching and learning is more than the mere acquisition of content. It represents a process of learning by thinking-do-thinking.info:eu-repo/semantics/publishedVersio

Are Games All Child’s Play?

The popularity of entertainment gaming over the last decades has led to the use of games for non-entertainment purposes in areas such as training and business support. The emergence of the serious games movement has capitalized on this interest in leisure gaming, with an increase in leisure game approaches in schools, colleges, universities and in professional training and continuing professional development. The movement raises many significant issues and challenges for us. How can gaming and simulation technologies be used to engage learners? How can games be used to motivate, deepen and accelerate learning? How can they be used to greatest effect in learning and teaching? The contributors explore these and many other questions that are vital to our understanding of the paradigm shift from conventional learning environments to learning in games and simulations

An action selection architecture for autonomous virtual humans in persistent worlds

Nowadays, virtual humans such as non-player characters in computer games need to have a strong autonomy in order to live their own life in persistent virtual worlds. When designing autonomous virtual humans, the action selection problem needs to be considered, as it is responsible for decision making at each moment in time. Indeed action selection architectures for autonomous virtual humans need to be reactive, proactive, motivational, and emotional to obtain a high degree of autonomy and individuality. The thesis can be divided into three parts. In the first part, we define each word of our title to precise their sense and raise the problematic of this work. We describe also inspirations from several domains that we used to design our model because this thesis is highly multi-disciplinary. Indeed, decision-making is essential for every autonomous entity and is studied in ethology, robotics, computer graphics, computer sciences, and cognitive sciences. However, we have chosen specific techniques to implement our model: hierarchical classifier systems and a free flow hierarchy. The second part of this thesis describes in detail our model of action selection for autonomous virtual humans. We use overlapping hierarchical classifier systems, working in parallel, to generate coherent behavioral plans. They are associated with the functionalities of a free flow hierarchy for the spreading of activation to give reactivity and flexibility to the hierarchical system. Moreover several functionalities are added to enhance and facilitate the choice of the most appropriate action at every time according to the internal and external influences. Finally, in the third part of this thesis, a complex simulated environment is created for testing the model and its functionalities with many conflicting motivations. Results demonstrate that the model is sufficiently efficient, robust and flexible for designing motivational autonomous virtual humans in persistent worlds. Moreover, we have just started to investigate on the emotional level which has to be improved in the future to have more subjective and adaptive behaviors and also manage social interactions with other virtual humans or users. Applied to video games, non player characters are more interesting and believable because they live their own life when people don't interact with them

Mastery and the mobile future of massively multiplayer games

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Comparative Media Studies, 2007.Includes bibliographical references (p. 64-66).What game design opportunities do we create when we extend massively multiplayer online games (MMOs) to cell phones? MMOs allow us to create representations of our own increasing mastery, and mobile gives us better access to this mastery and allows us to integrate it more fully into the ways we see ourselves. MMOs motivate mastery by making that mastery personally and socially relevant, and visibly showing it increase. Virtual worlds that make players feel physically and socially present increase motivation to achieve mastery. MMOs that convince players their avatars represent some aspect of their personalities increase motivation to invest in and experiment with different constructions of self. I apply these principles to an analysis of two games: Labyrinth, a game I helped create, and World of Warcraft, the current leading MMO. With Labyrinth, I explain the design decisions we made and their impact. With World of Warcraft, I described how altering the design could accommodate mobile play and better motivate increasing mastery.by Daniel Roy.S.M

Profiling the educational value of computer games

There are currently a number of suggestions for educators to include computer games in formal teaching and learning contexts. Educational value is based on claims that games promote the development of complex learning. Very little research, however, has explored what features should be present in a computer game to make it valuable or conducive to learning. We present a list of required features for an educational game to be of value, informed by two studies, which integrated theories of Learning Environments and Learning Styles. A user survey showed that some requirements were typical of games in a particular genre, while other features were present across all genres. The paper concludes with a proposed framework of games and features within and across genres to assist in the design and selection of games for a given educational scenari

Spontaneous Communities of Learning: Cooperative Learning Ecosystems Surrounding Virtual Worlds

This thesis is the culmination of a five year research project exploring online gamers and the cultures they engage with, both virtually in the many massively multiplayer games and virtual worlds online, and in the physical spaces they inhabit in various play spaces around the world. The primary research questions concerned social learning in such spaces, i.e. how do players learn from one another what they need to be successful, and what are the associated norms and practices for doing so? What sorts of peripheral skills are gained, and are they applicable to physical world contexts? Finally, what does participation in such spaces mean for individuals who may have lacked other mechanisms for social learning, and what impacts might such findings have on existing educational structures? I anticipate that this thesis will generate as many questions as it will answer, and I hope, that as a snapshot of a gaming culture in time, will be looked upon as a monograph in the classic ethnographic tradition