Assessing the Potential of Classical Q-learning in General Game Playing

After the recent groundbreaking results of AlphaGo and AlphaZero, we have seen strong interests in deep reinforcement learning and artificial general intelligence (AGI) in game playing. However, deep learning is resource-intensive and the theory is not yet well developed. For small games, simple classical table-based Q-learning might still be the algorithm of choice. General Game Playing (GGP) provides a good testbed for reinforcement learning to research AGI. Q-learning is one of the canonical reinforcement learning methods, and has been used by (Banerjee $\&$ Stone, IJCAI 2007) in GGP. In this paper we implement Q-learning in GGP for three small-board games (Tic-Tac-Toe, Connect Four, Hex)\footnote{source code: https://github.com/wh1992v/ggp-rl}, to allow comparison to Banerjee et al.. We find that Q-learning converges to a high win rate in GGP. For the $\epsilon$-greedy strategy, we propose a first enhancement, the dynamic $\epsilon$ algorithm. In addition, inspired by (Gelly $\&$ Silver, ICML 2007) we combine online search (Monte Carlo Search) to enhance offline learning, and propose QM-learning for GGP. Both enhancements improve the performance of classical Q-learning. In this work, GGP allows us to show, if augmented by appropriate enhancements, that classical table-based Q-learning can perform well in small games.Comment: arXiv admin note: substantial text overlap with arXiv:1802.0594

Assessing the Potential of Classical Q-learning in General Game Playing

After the recent groundbreaking results of AlphaGo and AlphaZero, we have seen strong interests in deep reinforcement learning and artificial general intelligence (AGI) in game playing. However, deep learning is resource-intensive and the theory is not yet well developed. For small games, simple classical table-based Q-learning might still be the algorithm of choice. General Game Playing (GGP) provides a good testbed for reinforcement learning to research AGI. Q-learning is one of the canonical reinforcement learning methods, and has been used by (Banerjee & Stone, IJCAI 2007) in GGP. In this paper we implement Q-learning in GGP for three small-board games (Tic-Tac-Toe, Connect Four, Hex), to allow comparison to Banerjee et al. We find that Q-learning converges to a high win rate in GGP. For the ϵ" role="presentation" style="display: inline-table; line-height: normal; letter-spacing: normal; word-spacing: normal; overflow-wrap: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px; border-width: 0px; border-style: initial; position: relative;">ϵ-greedy strategy, we propose a first enhancement, the dynamic ϵ" role="presentation" style="display: inline-table; line-height: normal; letter-spacing: normal; word-spacing: normal; overflow-wrap: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px; border-width: 0px; border-style: initial; position: relative;">ϵ algorithm. In addition, inspired by (Gelly & Silver, ICML 2007) we combine online search (Monte Carlo Search) to enhance offline learning, and propose QM-learning for GGP. Both enhancements improve the performance of classical Q-learning. In this work, GGP allows us to show, if augmented by appropriate enhancements, that classical table-based Q-learning can perform well in small games.Computer Systems, Imagery and Medi

Understanding the Cultural Value of 'In Harmony-Sistema England'

This research project on which this paper reports was designed to explore questions of cultural value in relation to the schools music project In Harmony-Sistema England. Our core research focus has been upon the ways in which children, their teachers and tutors, and their families understand the value of their participation in IHSE initiatives. The project engaged with three case studies of IHSE initiatives (based in Norwich, Telford and Newcastle) and qualitative data was gathered with primary school children, school staff, parents and IHSE musicians in all three cases

Unmasking Clever Hans Predictors and Assessing What Machines Really Learn

Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly "intelligent" behavior. Here we apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This showcases a spectrum of problem-solving behaviors ranging from naive and short-sighted, to well-informed and strategic. We observe that standard performance evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors. Furthermore, we propose our semi-automated Spectral Relevance Analysis that provides a practically effective way of characterizing and validating the behavior of nonlinear learning machines. This helps to assess whether a learned model indeed delivers reliably for the problem that it was conceived for. Furthermore, our work intends to add a voice of caution to the ongoing excitement about machine intelligence and pledges to evaluate and judge some of these recent successes in a more nuanced manner.Comment: Accepted for publication in Nature Communication

Internet and gaming addiction: a systematic literature review of neuroimaging studies

In the past decade, research has accumulated suggesting that excessive Internet use can lead to the development of a behavioral addiction. Internet addiction has been considered as a serious threat to mental health and the excessive use of the Internet has been linked to a variety of negative psychosocial consequences. The aim of this review is to identify all empirical studies to date that used neuroimaging techniques to shed light upon the emerging mental health problem of Internet and gaming addiction from a neuroscientific perspective. Neuroimaging studies offer an advantage over traditional survey and behavioral research because with this method, it is possible to distinguish particular brain areas that are involved in the development and maintenance of addiction. A systematic literature search was conducted, identifying 18 studies. These studies provide compelling evidence for the similarities between different types of addictions, notably substance-related addictions and Internet and gaming addiction, on a variety of levels. On the molecular level, Internet addiction is characterized by an overall reward deficiency that entails decreased dopaminergic activity. On the level of neural circuitry, Internet and gaming addiction led to neuroadaptation and structural changes that occur as a consequence of prolonged increased activity in brain areas associated with addiction. On a behavioral level, Internet and gaming addicts appear to be constricted with regards to their cognitive functioning in various domains. The paper shows that understanding the neuronal correlates associated with the development of Internet and gaming addiction will promote future research and will pave the way for the development of addiction treatment approaches

Evolutionary games on graphs

Game theory is one of the key paradigms behind many scientific disciplines from biology to behavioral sciences to economics. In its evolutionary form and especially when the interacting agents are linked in a specific social network the underlying solution concepts and methods are very similar to those applied in non-equilibrium statistical physics. This review gives a tutorial-type overview of the field for physicists. The first three sections introduce the necessary background in classical and evolutionary game theory from the basic definitions to the most important results. The fourth section surveys the topological complications implied by non-mean-field-type social network structures in general. The last three sections discuss in detail the dynamic behavior of three prominent classes of models: the Prisoner's Dilemma, the Rock-Scissors-Paper game, and Competing Associations. The major theme of the review is in what sense and how the graph structure of interactions can modify and enrich the picture of long term behavioral patterns emerging in evolutionary games.Comment: Review, final version, 133 pages, 65 figure

Using a gamified monitoring app to change adolescents' snack intake : the development of the REWARD app and evaluation design

Background: As the snacking pattern of European adolescents is of great concern, effective interventions are necessary. Till now health promotion efforts in children and adolescents have had only limited success in changing adolescents' eating patterns and anthropometrics. Therefore, the present study proposes an innovative approach to influence dietary behaviors in youth based on new insights on effective behavior change strategies and attractive intervention channels to engage adolescents. This article describes the rationale, the development, and evaluation design of the 'Snack Track School' app. The aim of the app is to improve the snacking patterns of Flemish 14- to 16-year olds. Methods: The development of the app was informed by the systematic, stepwise, iterative, and collaborative principles of the Intervention Mapping protocol. A four week mHealth intervention was developed based on the dual-system model with behavioral change strategies targeting both the reflective (i.e., active learning, advance organizers, mere exposure, goal-setting, monitoring, and feedback) and automatic processes (i.e., rewards and positive reinforcement). This intervention will be evaluated via a controlled pre-post design in Flemish schools among 1400 adolescents. Discussion: When this intervention including strategies focused on both the reflective and automatic pathway proves to be effective, it will offer a new scientifically-based vision, guidelines and practical tools for public health and health promotion (i.e., incorporation of learning theories in intervention programs)

How agency models inspire large scale participatory planning and its evaluation

International audienceWe describe how three models, for sustainable change, human agency in collective resource management, and socio-environmental systems, have been used to design a protocol and the tools for a large scale (1500 participants, 35 villages) multi-level participatory process held in Africa for Integrated Natural Resource Management, through the European Project Afromaison. The process especially combines a common action model to support proposals by stakeholders, an integration matrix to build coherent plans, a role playing game design process, and a method to combine planning and playing to engage into the plans. It has also inspired the design of the attached monitoring and evaluation process. We describe the process in two countries, Ethiopia and Uganda, present the theoretical bases of the evaluation framework using the ENCORE paradigm and the implemented methodology transferred to local evaluators. We introduce some results and propose comments on potential learning back to the modelling community