Resolving social dilemmas with minimal reward transfer

Multi-agent cooperation is an important topic, and is particularly challenging in mixed-motive situations where it does not pay to be nice to others. Consequently, self-interested agents often avoid collective behaviour, resulting in suboptimal outcomes for the group. In response, in this paper we introduce a metric to quantify the disparity between what is rational for individual agents and what is rational for the group, which we call the general self-interest level. This metric represents the maximum proportion of individual rewards that all agents can retain while ensuring that achieving social welfare optimum becomes a dominant strategy. By aligning the individual and group incentives, rational agents acting to maximise their own reward will simultaneously maximise the collective reward. As agents transfer their rewards to motivate others to consider their welfare, we diverge from traditional concepts of altruism or prosocial behaviours. The general self-interest level is a property of a game that is useful for assessing the propensity of players to cooperate and understanding how features of a game impact this. We illustrate the effectiveness of our method on several novel games representations of social dilemmas with arbitrary numbers of players.Comment: 34 pages, 13 tables, submitted to the Journal of Autonomous Agents and Multi-Agent Systems: Special Issue on Citizen-Centric AI System

Cooperation and Social Dilemmas with Reinforcement Learning

Cooperation between humans has been foundational for the development of civilisation and yet there are many questions about how it emerges from social interactions. As artificial agents begin to play a more significant role in our lives and are introduced into our societies, it is apparent that understanding the mechanisms of cooperation is important also for the design of next-generation multi-agent AI systems. Indeed, this is particularly important in the case of supporting cooperation between self-interested AI agents. In this thesis, we focus on the analysis of the application of mechanisms that are at the basis of human cooperation to the training of reinforcement learning agents. Human behaviour is a product of cultural norms, emotions and intuition amongst other things: we argue it is possible to use similar mechanisms to deal with the complexities of multi-agent cooperation. We outline the problem of cooperation in mixed-motive games, also known as social dilemmas, and we focus on the mechanisms of reputation dynamics and partner selection, two mechanisms that have been strongly linked to indirect reciprocity in Evolutionary Game Theory. A key point that we want to emphasise is the fact we assume no prior knowledge and explicit definition of strategies, which instead are fully learnt by the agents during the games. In our experimental evaluation, we demonstrate the benefits of applying these mechanisms to the training process of the agents, and we compare our findings with results presented in a variety of other disciplines, including Economics and Evolutionary Biology

Interactive Narrative for Adaptive Educational Games: Architecture and an Application to Character Education

This thesis presents AEINS, Adaptive Educational Interactive Narrative System, that supports teaching ethics for 8-12 year old children. AEINS is designed based on Keller's and Gagné's learning theories. The idea is centered around involving students in moral dilemmas (called teaching moments) within which the Socratic Method is used as the teaching pedagogy. The important unique aspect of AEINS is that it exhibits the presence of four features shown to individually increase effectiveness of edugames environments, yet not integrated together in past research: a student model, a dynamic generated narrative, scripted branched narrative and evolving non-player characters. The student model aims to provide adaptation. The dynamic generated narrative forms a continuous story that glues the scripted teaching moments together. The evolving agents increase the realism and believability of the environment and perform a recognized pedagogical role by helping in supplying the educational process. AEINS has been evaluated intrinsically and empirically according to the following themes: architecture and implementation, social aspects, and educational achievements. The intrinsic evaluation checked the implicit goals embodied by the design aspects and made a value judgment about these goals. In the empirical evaluation, twenty participants were assigned to use AEINS over a number of games. The evaluation showed positive results as the participants appreciated the social characteristics of the system as they were able to recognize the genuine social aspects and the realism represented in the game. Finally, the evaluation showed indications for developing new lines of thinking for some participants to the extent that some of them were ready to carry the experience forward to the real world. However, the evaluation also suggested possible improvements, such as the use of 3D interface and free text natural language

Leading Digital Socio-Economy to Efficiency -- A Primer on Tokenomics

Through the usage of cryptographically secure and digitally scarce tokens, a next evolutionary step of the Internet has come. Cryptographic token represent a new phenomena of the crypto movement with the ability to program rules and incentives to steer participant behaviors that transforms them from purely technical to socio-economic innovations. Tokens allow the coordination, optimization and governing large networks of resources in a decentralized manner and at scale. Tokens bring powerful network effects that reward participants relative to their stage of adoption, the value they contribute and the risk they bear in an auditable, decentralized and therefore trustful way. We illustrate which important role tokenomics plays in the creation of, and sustainable and efficient operation of cooperations.Comment: 9 page

Towards a million change agents. A review of the social movements literature: implications for large scale change in the NHS.

This review explores 'social movements' as a new way of thinking about large-scale systems change and assesses the potential contribution of applying this new perspective to NHS improvement

Cooperative artificial intelligence

In the future, artificial learning agents are likely to become increasingly widespread in our society. They will interact with both other learning agents and humans in a variety of complex settings including social dilemmas. We argue that there is a need for research on the intersection between game theory and artificial intelligence, with the goal of achieving cooperative artificial intelligence that can navigate social dilemmas well. We consider the problem of how an external agent can promote cooperation between artificial learners by distributing additional rewards and punishments based on observing the learners' actions. We propose a rule for automatically learning how to create right incentives by considering the players' anticipated parameter updates. Using this learning rule leads to cooperation with high social welfare in matrix games in which the agents would otherwise learn to defect with high probability. We show that the resulting cooperative outcome is stable in certain games even if the planning agent is turned off after a given number of episodes, while other games require ongoing intervention to maintain mutual cooperation. Finally, we reflect on what the goals of multi-agent reinforcement learning should be in the first place, and discuss the necessary building blocks towards the goal of building cooperative AI

Green political theory: nature, virtue and progress

This thesis offers an immanent critique and reconstruction of green moral and political theory. In chapter 1, the critical-reconstructive approach and spirit of the thesis is outlined in terms of contributing to the process of developing a green political theory that is different from 'ecologism' or ideological accounts of green politics. In chapter 2, deep ecology is critically interrogated in terms of its metaphysical (2.3) and psychological claims (2.4). Its view of the 'ecological crisis' as a 'crisis' of western culture is criticised as is its a priori defence of environmental preservation over the human productive use of nature. While its ecocentrism is rejected as the normative basis for green politics, its concern with virtue ethics is held to be an important contribution. In chapter 3, a self-reflexive version of anthropocentrism is developed as the most appropriate moral basis for green politics. Some naturalistic arguments are presented in order to support 'speciesism', and defend it from claims of arbitrariness and as being akin to sexism or racism. Arguments centring on demonstrating the tenuous character of the differences between humans and nonhumans are argued to neglect the fundamental moral significance of the difference between 'human' and 'nonhuman'. I argue that an ethic of use, understood as a reflexive mode of interaction with the nonhuman world, is a defensible form of anthropocentrism for green political purposes. The basis of this reflexive anthropocentrism turns on the claim that while human interests are a necessary condition for justifying a particular human use of nature, it is not a satisfactory one. Issues pertaining to the 'seriousness' of the human interest which is fulfilled are held to be important in distinguishing 'use' from 'abuse'

Routing choices in intelligent transport systems

Road congestion is a phenomenon that can often be avoided; roads become popular, travel times increase, which could be mitigated with better coordination mechanisms. The choice of route, mode of transport, and departure time all play a crucial part in controlling congestion levels. Technology, such as navigation applications, have the ability to influence these decisions and play an essential role in congestion reduction. To predict vehicles' routing behaviours, we model the system as a game with rational players. Players choose a path between origin and destination nodes in a network. Each player seeks to minimise their own journey time, often leading to inefficient equilibria with poor social welfare. Traffic congestion motivates the results in this thesis. However, the results also hold true for many other applications where congestion occurs, e.g. power grid demand. Coordinating route selection to reduce congestion constitutes a social dilemma for vehicles. In sequential social dilemmas, players' strategies need to balance their vulnerability to exploitation from their opponents and to learn to cooperate to achieve maximal payouts. We address this trade-off between mathematical safety and cooperation of strategies in social dilemmas to motivate our proposed algorithm, a safe method of achieving cooperation in social dilemmas, including route choice games. Many vehicles use navigation applications to help plan their journeys, but these provide only partial information about the routes available to them. We find a class of networks for which route information distribution cannot harm the receiver's expected travel times. Additionally, we consider a game where players always follow the route chosen by an application or where vehicle route selection is controlled by a route planner, such as autonomous vehicles. We show that having multiple route planners controlling vehicle routing leads to inefficient equilibria. We calculate the Price of Anarchy (PoA) for polynomial function travel times and show that multiagent reinforcement learning algorithms suffer from the predicted Price of Anarchy when controlling vehicle routing. Finally, we equip congestion games with waiting times at junctions to model the properties of traffic lights at intersections. Here, we show that Braess' paradox can be avoided by implementing traffic light cycles and establish the PoA for realistic waiting times. By employing intelligent traffic lights that use myopic learning, such as multi-agent reinforcement learning, we prove a natural reward function guarantees convergence to equilibrium. Moreover, we highlight the impact of multi-agent reinforcement learning traffic lights on the fairness of journey times to vehicles