A conceptual framework for externally-influenced agents: an assisted reinforcement learning review

A long-term goal of reinforcement learning agents is to be able to perform tasks in complex real-world scenarios. The use of external information is one way of scaling agents to more complex problems. However, there is a general lack of collaboration or interoperability between different approaches using external information. In this work, while reviewing externally-influenced methods, we propose a conceptual framework and taxonomy for assisted reinforcement learning, aimed at fostering collaboration by classifying and comparing various methods that use external information in the learning process. The proposed taxonomy details the relationship between the external information source and the learner agent, highlighting the process of information decomposition, structure, retention, and how it can be used to influence agent learning. As well as reviewing state-of-the-art methods, we identify current streams of reinforcement learning that use external information in order to improve the agent’s performance and its decision-making process. These include heuristic reinforcement learning, interactive reinforcement learning, learning from demonstration, transfer learning, and learning from multiple sources, among others. These streams of reinforcement learning operate with the shared objective of scaffolding the learner agent. Lastly, we discuss further possibilities for future work in the field of assisted reinforcement learning systems. © 2021, The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature

Agent Street: An Environment for Exploring Agent-Based Models in Second Life

Urban models can be seen on a continuum between iconic and symbolic. Generally speaking, iconic models are physical versions of the real world at some scaled down representation, while symbolic models represent the system in terms of the way they function replacing the physical or material system by some logical and/or mathematical formulae. Traditionally iconic and symbolic models were distinct classes of model but due to the rise of digital computing the distinction between the two is becoming blurred, with symbolic models being embedded into iconic models. However, such models tend to be single user. This paper demonstrates how 3D symbolic models in the form of agent-based simulations can be embedded into iconic models using the multi-user virtual world of Second Life. Furthermore, the paper demonstrates Second Life\'s potential for social science simulation. To demonstrate this, we first introduce Second Life and provide two exemplar models; Conway\'s Game of Life, and Schelling\'s Segregation Model which highlight how symbolic models can be viewed in an iconic environment. We then present a simple pedestrian evacuation model which merges the iconic and symbolic together and extends the model to directly incorporate avatars and agents in the same environment illustrating how \'real\' participants can influence simulation outcomes. Such examples demonstrate the potential for creating highly visual, immersive, interactive agent-based models for social scientists in multi-user real time virtual worlds. The paper concludes with some final comments on problems with representing models in current virtual worlds and future avenues of research.Agent-Based Modelling, Pedestrian Evacuation, Segregation, Virtual Worlds, Second Life

How social learning strategies boost or undermine decision making in groups

Social interactions resulting in emergent collective behaviour play a key role in almost all layers of society, from local, small-scale interactions, such as people crossing the street, to global, large-scale interactions, such as the spread of fake news on online platforms. In our digital and interconnected world, it is increasingly important to understand the emergence of beneficial or detrimental collective dynamics. The characteristics of such dynamics are expected to depend greatly on the nature of information individuals have personally acquired and how they learn from others. Yet, how the decision-making processes shape the resulting collective dynamics remains poorly understood. When do individuals seek more information from social sources? How do individuals reap the benefits when navigating in social environments, and when do they fail to do so? This dissertation aims to answer these questions extending established theories and frameworks from individual decision-making into the social realm. This approach allows for the operationalization of personal and social information in a theory-driven manner, thereby achieving a deeper understanding of the individual-level decision process. The first chapter provides a introductory overview of the interplay between personal information use, social learning strategies and collective dynamics, and introduces the key theories and models I will expand on in this dissertation. In Chapter 2, inspired by Brunswick's lens model, I investigate how individuals form beliefs about the meaning of ecological structures (i.e., cues). Here, participants had to categorize images based on multiple cues, the meaning of which had to be learned over trials. I showed that participants observing the same cues formed different beliefs about the cue meanings. This diversity in cue beliefs is, in turn, an important process governing the quality of social information. The greater this diversity, the more independent personal information is, and the stronger the potential for social information use. Participants, however, failed to realize the full potential of this diversity because they only changed their personal decisions if a large majority disagreed with them. Simulating different strategies of social information use, I show that this reliance on strongly agreeing majorities impedes individuals from benefiting from diversity. This chapter thus identifies diversity in cue beliefs as an important factor allowing individuals in groups to benefit from the wisdom of each other, while simultaneously highlighting the importance of the individuals' social learning strategies to exploit this diversity. Chapter 3 dives deeper into the social learning strategies individuals use. By carefully controlling the social information displayed to participants, the study in this chapter provides an in-depth analysis of social learning strategies. Participants were confronted with an estimation task. They first provided an independent estimate, after which they observed estimates of others. Using Bayesian modelling techniques, I show that the incorporation of others' opinions strongly depends on how consistent those opinions are with an individual's own opinion and the degree of agreement among others. Individuals also strongly differ in the social learning strategies they use. These results elucidate what aspects are conducive for people to change their minds and contribute to the understanding of how individuals’ social information use shapes opinion and information dynamics in our interconnected society. In Chapter 4, I embed individuals a in temporal dynamic system which allows the investigation of the use of information in interaction with the emergent collective dynamic. Here, my focus is on social interactions where multiple people make decisions sequentially and thereby are simultaneously emitters and receivers of social information. To shed light on the unfolding dynamic in such settings, I will introduce the social drift-diffusion model (DDM). The model allows the investigation of the cognitive processes underlying the integration of personal and social information dynamically over time, and the subsequent collective dynamic. Analysis of the data shows that correct information spreads when the participants’ confidence reflects accuracy and when more confident participants decide faster. Under these conditions, later-deciding participants are likely to adopt social information and thereby to amplify the correct signal provided by early-deciding participants. The social DDM successfully captures all the key dynamics observed in the social system, revealing the cognitive underpinnings of information cascades in social systems. The general principles of personal and social information use that emerge from Chapter 4 allow to investigate the optimal behaviour when deciding sequentially. In Chapter 5, I develop an agent-based version of the social DDM and embed it in evolutionary algorithms, allowing the identification of evolutionarily advantageous strategies. I show that the individuals' decision time should depend on the quality of information, with the most accurate individuals deciding first. For all later.deciding individuals it is evolutionary advantageous to imitate the (often accurate) first decision. When introducing asymmetric error costs, single individuals should develop response biases to avoid the more costly error. In groups, however, such response biases can have dramatic consequences, as these biases are likely to be amplified in the group. As a result, individuals in large groups should use much weaker response biases to benefit from social information. I conclude that individuals facing asymmetric error costs in social environments need to carefully trade off the expressed response bias and sensitivity to social information to avoid the more costly error but simultaneously benefit from the collective. Overall, this thesis deepens our understanding of social dynamics by accounting for individual-level decision-making processes across various choice problems. I show that the behaviour of individuals in social environments can significantly differ depending on the personal information individuals possess and the strategies individuals use. Furthermore, I highlight the importance of accounting for such differences to predict the emergence of beneficial or detrimental dynamics in social environments