New And Surprising Ways to Be Mean: Adversarial NPCs with Coupled Empowerment Minimisation

Creating Non-Player Characters (NPCs) that can react robustly to unforeseen player behaviour or novel game content is difficult and time-consuming. This hinders the design of believable characters, and the inclusion of NPCs in games that rely heavily on procedural content generation. We have previously addressed this challenge by means of empowerment, a model of intrinsic motivation, and demonstrated how a coupled empowerment maximisation (CEM) policy can yield generic, companion-like behaviour. In this paper, we extend the CEM framework with a minimisation policy to give rise to adversarial behaviour. We conduct a qualitative, exploratory study in a dungeon-crawler game, demonstrating that CEM can exploit the affordances of different content facets in adaptive adversarial behaviour without modifications to the policy. Changes to the level design, underlying mechanics and our character's actions do not threaten our NPC's robustness, but yield new and surprising ways to be mean

Intrinsic Motivation in Computational Creativity Applied to Videogames

PhD thesisComputational creativity (CC) seeks to endow artificial systems with creativity. Although human creativity is known to be substantially driven by intrinsic motivation (IM), most CC systems are extrinsically motivated. This restricts their actual and perceived creativity and autonomy, and consequently their benefit to people. In this thesis, we demonstrate, via theoretical arguments and through applications in videogame AI, that computational intrinsic reward and models of IM can advance core CC goals. We introduce a definition of IM to contextualise related work. Via two systematic reviews, we develop typologies of the benefits and applications of intrinsic reward and IM models in CC and game AI. Our reviews highlight that related work is limited to few reward types and motivations, and we thus investigate the usage of empowerment, a little studied, information-theoretic intrinsic reward, in two novel models applied to game AI. We define coupled empowerment maximisation (CEM), a social IM model, to enable general co-creative agents that support or challenge their partner through emergent behaviours. Via two qualitative, observational vignette studies on a custom-made videogame, we explore CEM’s ability to drive general and believable companion and adversary non-player characters which respond creatively to changes in their abilities and the game world. We moreover propose to leverage intrinsic reward to estimate people’s experience of interactive artefacts in an autonomous fashion. We instantiate this proposal in empowerment-based player experience prediction (EBPXP) and apply it to videogame procedural content generation. By analysing think-aloud data from an experiential vignette study on a dedicated game, we identify several experiences that EBPXP could predict. Our typologies serve as inspiration and reference for CC and game AI researchers to harness the benefits of IM in their work. Our new models can increase the generality, autonomy and creativity of next-generation videogame AI, and of CC systems in other domains

Action Selection in the Creative Systems Framework

The Creative Systems Framework (CSF) formalises creativity as search through a space of concepts. As a formal account of Margaret Boden’s descriptive hierarchy of creativity, it is at the basis of multiple studies dealing with diverse aspects of Computational Creativity (CC) systems. However, the CSF at present neither formalises action nor action selection during search, limiting its use in analysing creative processes. We extend the CSF by explicitly modelling these missing components in the search space traversal function. We furthermore introduce a distinction between a concept and its material realisation as an artefact, and elaborate the action selection process to provide stopping criteria for creative search. Our extension, the Creative Action Selection Framework (CASF), is informed by previous studies in CC and draws on concepts from Markov Decision Processes (MDP). It allows us to describe a creative system as an agent selecting actions based on the value, validity and novelty of concepts and artefacts. The CASF brings the descriptive power of the CSF to a wider range of systems with more analytical depth.Peer reviewe

Expanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop

Active inference is an ambitious theory that treats perception, inference, and action selection of autonomous agents under the heading of a single principle. It suggests biologically plausible explanations for many cognitive phenomena, including consciousness. In active inference, action selection is driven by an objective function that evaluates possible future actions with respect to current, inferred beliefs about the world. Active inference at its core is independent from extrinsic rewards, resulting in a high level of robustness across e.g., different environments or agent morphologies. In the literature, paradigms that share this independence have been summarized under the notion of intrinsic motivations. In general and in contrast to active inference, these models of motivation come without a commitment to particular inference and action selection mechanisms. In this article, we study if the inference and action selection machinery of active inference can also be used by alternatives to the originally included intrinsic motivation. The perception-action loop explicitly relates inference and action selection to the environment and agent memory, and is consequently used as foundation for our analysis. We reconstruct the active inference approach, locate the original formulation within, and show how alternative intrinsic motivations can be used while keeping many of the original features intact. Furthermore, we illustrate the connection to universal reinforcement learning by means of our formalism. Active inference research may profit from comparisons of the dynamics induced by alternative intrinsic motivations. Research on intrinsic motivations may profit from an additional way to implement intrinsically motivated agents that also share the biological plausibility of active inference

Reinforcement learning in large state action spaces

Reinforcement learning (RL) is a promising framework for training intelligent agents which learn to optimize long term utility by directly interacting with the environment. Creating RL methods which scale to large state-action spaces is a critical problem towards ensuring real world deployment of RL systems. However, several challenges limit the applicability of RL to large scale settings. These include difficulties with exploration, low sample efficiency, computational intractability, task constraints like decentralization and lack of guarantees about important properties like performance, generalization and robustness in potentially unseen scenarios. This thesis is motivated towards bridging the aforementioned gap. We propose several principled algorithms and frameworks for studying and addressing the above challenges RL. The proposed methods cover a wide range of RL settings (single and multi-agent systems (MAS) with all the variations in the latter, prediction and control, model-based and model-free methods, value-based and policy-based methods). In this work we propose the first results on several different problems: e.g. tensorization of the Bellman equation which allows exponential sample efficiency gains (Chapter 4), provable suboptimality arising from structural constraints in MAS(Chapter 3), combinatorial generalization results in cooperative MAS(Chapter 5), generalization results on observation shifts(Chapter 7), learning deterministic policies in a probabilistic RL framework(Chapter 6). Our algorithms exhibit provably enhanced performance and sample efficiency along with better scalability. Additionally, we also shed light on generalization aspects of the agents under different frameworks. These properties have been been driven by the use of several advanced tools (e.g. statistical machine learning, state abstraction, variational inference, tensor theory). In summary, the contributions in this thesis significantly advance progress towards making RL agents ready for large scale, real world applications

New and Surprising Ways to Be Mean

Creating Non-Player Characters (NPCs) that can react robustly to unforeseen player behaviour or novel game content is difficult and time-consuming. This hinders the design of believable characters, and the inclusion of NPCs in games that rely heavily on procedural content generation. We have previously addressed this challenge by means of empowerment, a model of intrinsic motivation, and demonstrated how a coupled empowerment maximisation (CEM) policy can yield generic, companion-like behaviour. In this paper, we extend the CEM framework with a minimisation policy to give rise to adversarial behaviour. We conduct a qualitative, exploratory study in a dungeon-crawler game, demonstrating that CEM can exploit the affordances of different content facets in adaptive adversarial behaviour without modifications to the policy. Changes to the level design, underlying mechanics and our character's actions do not threaten our NPC's robustness, but yield new and surprising ways to be mean