On Partially Controlled Multi-Agent Systems

Motivated by the control theoretic distinction between controllable and uncontrollable events, we distinguish between two types of agents within a multi-agent system: controllable agents, which are directly controlled by the system's designer, and uncontrollable agents, which are not under the designer's direct control. We refer to such systems as partially controlled multi-agent systems, and we investigate how one might influence the behavior of the uncontrolled agents through appropriate design of the controlled agents. In particular, we wish to understand which problems are naturally described in these terms, what methods can be applied to influence the uncontrollable agents, the effectiveness of such methods, and whether similar methods work across different domains. Using a game-theoretic framework, this paper studies the design of partially controlled multi-agent systems in two contexts: in one context, the uncontrollable agents are expected utility maximizers, while in the other they are reinforcement learners. We suggest different techniques for controlling agents' behavior in each domain, assess their success, and examine their relationship.Comment: See http://www.jair.org/ for any accompanying file

Combining Subgoal Graphs with Reinforcement Learning to Build a Rational Pathfinder

In this paper, we present a hierarchical path planning framework called SG-RL (subgoal graphs-reinforcement learning), to plan rational paths for agents maneuvering in continuous and uncertain environments. By "rational", we mean (1) efficient path planning to eliminate first-move lags; (2) collision-free and smooth for agents with kinematic constraints satisfied. SG-RL works in a two-level manner. At the first level, SG-RL uses a geometric path-planning method, i.e., Simple Subgoal Graphs (SSG), to efficiently find optimal abstract paths, also called subgoal sequences. At the second level, SG-RL uses an RL method, i.e., Least-Squares Policy Iteration (LSPI), to learn near-optimal motion-planning policies which can generate kinematically feasible and collision-free trajectories between adjacent subgoals. The first advantage of the proposed method is that SSG can solve the limitations of sparse reward and local minima trap for RL agents; thus, LSPI can be used to generate paths in complex environments. The second advantage is that, when the environment changes slightly (i.e., unexpected obstacles appearing), SG-RL does not need to reconstruct subgoal graphs and replan subgoal sequences using SSG, since LSPI can deal with uncertainties by exploiting its generalization ability to handle changes in environments. Simulation experiments in representative scenarios demonstrate that, compared with existing methods, SG-RL can work well on large-scale maps with relatively low action-switching frequencies and shorter path lengths, and SG-RL can deal with small changes in environments. We further demonstrate that the design of reward functions and the types of training environments are important factors for learning feasible policies.Comment: 20 page

Psychological factors affecting equine performance

For optimal individual performance within any equestrian discipline horses must be in peak physical condition and have the correct psychological state. This review discusses the psychological factors that affect the performance of the horse and, in turn, identifies areas within the competition horse industry where current behavioral research and established behavioral modification techniques could be applied to further enhance the performance of animals. In particular, the role of affective processes underpinning temperament, mood and emotional reaction in determining discipline-specific performance is discussed. A comparison is then made between the training and the competition environment and the review completes with a discussion on how behavioral modification techniques and general husbandry can be used advantageously from a performance perspective

Stress and Decision Making: Effects on Valuation, Learning, and Risk-taking

A wide range of stressful experiences can influence human decision making in complex ways beyond the simple predictions of a fight-or-flight model. Recent advances may provide insight into this complicated interaction, potentially in directions that could result in translational applications. Early research suggests that stress exposure influences basic neural circuits involved in reward processing and learning, while also biasing decisions toward habit and modulating our propensity to engage in risk-taking. That said, a substantial array of theoretical and methodological considerations in research on the topic challenge strong cross study comparisons necessary for the field to move forward. In this review we examine the multifaceted stress construct in the context of human decision making, emphasizing stress’ effect on valuation, learning, and risk-taking

Closing the loop between neural network simulators and the OpenAI Gym

Since the enormous breakthroughs in machine learning over the last decade, functional neural network models are of growing interest for many researchers in the field of computational neuroscience. One major branch of research is concerned with biologically plausible implementations of reinforcement learning, with a variety of different models developed over the recent years. However, most studies in this area are conducted with custom simulation scripts and manually implemented tasks. This makes it hard for other researchers to reproduce and build upon previous work and nearly impossible to compare the performance of different learning architectures. In this work, we present a novel approach to solve this problem, connecting benchmark tools from the field of machine learning and state-of-the-art neural network simulators from computational neuroscience. This toolchain enables researchers in both fields to make use of well-tested high-performance simulation software supporting biologically plausible neuron, synapse and network models and allows them to evaluate and compare their approach on the basis of standardized environments of varying complexity. We demonstrate the functionality of the toolchain by implementing a neuronal actor-critic architecture for reinforcement learning in the NEST simulator and successfully training it on two different environments from the OpenAI Gym

Does size really matter: a review of the role of stake and prize levels in relation to gambling-related harm

Regulatory and industry decisions influencing commercial gambling activities require clear understanding of the role that stakes and prizes play in the development and facilitation of gambling-related harm. Although industry proponents argue for increases in stakes and prizes to meet market demands, regulators remain cautious about the potential implication for gambling-related harm, while industry opponents generally condemn relaxing aspects of gambling policies. To inform this debate, this paper provides a critical examination of the relevant literature. From the review, it is concluded that limitations of the existing literature restrict our ability to draw definitive conclusions regarding the effects of stake and prize variables. Most studies contain multiple, methodological limitations, the most significant of which are diluted risk and reward scenarios used in analogue research settings not reflective of real gambling situations. In addition, there is a lack of conceptual clarity regarding many constructs, particularly the parameters defining jackpots, and the interactive nature and effect of the differing configurations of game parameters and environments are often not taken into consideration when investigating changes to one or more variables. Notwithstanding these limitations, there is sufficient evidence to suggest that stake and prize levels merit consideration in relation to harm minimisation efforts. However, substantial knowledge gaps currently exist, particularly in relation to understanding staking and prize thresholds for risky behaviour, how the impact of stakes and prizes change depending on the configuration and interaction of other game characteristics, and the role of individual and situational determinants. Based on the potential risk factors and the implications for commercial appeal, a player-focussed harm minimisation response may hold the most promise for future research and evaluation in jurisdictions where gambling is a legal and legitimate leisure activity