Decentralised Learning in Systems with Many, Many Strategic Agents

Although multi-agent reinforcement learning can tackle systems of strategically interacting entities, it currently fails in scalability and lacks rigorous convergence guarantees. Crucially, learning in multi-agent systems can become intractable due to the explosion in the size of the state-action space as the number of agents increases. In this paper, we propose a method for computing closed-loop optimal policies in multi-agent systems that scales independently of the number of agents. This allows us to show, for the first time, successful convergence to optimal behaviour in systems with an unbounded number of interacting adaptive learners. Studying the asymptotic regime of N-player stochastic games, we devise a learning protocol that is guaranteed to converge to equilibrium policies even when the number of agents is extremely large. Our method is model-free and completely decentralised so that each agent need only observe its local state information and its realised rewards. We validate these theoretical results by showing convergence to Nash-equilibrium policies in applications from economics and control theory with thousands of strategically interacting agents

Studying complex adaptive systems using molecular classifier systems

Complex Adaptive Systems (CAS) are dynamical networks of interacting agents occurring in a variety of natural and artificial systems (e.g. cells, societies, stock markets). These complex systems have the ability to adapt, evolve and learn from experience. To study CAS, Holland proposed to employ agent-based systems in which Learning Classifier Systems (LCS) are used to determine the agents behavior and adaptivity. We argue that LCS are limited for the study of CAS: the rule-discovery mechanism is pre-specified and may limit the evolvability of CAS. Secondly, LCS distinguish a demarcation between messages and rules, however operations are reflexive in CAS, e.g. in a cell, an agent (a molecule) may both act as a message (substrate) and as a catalyst (rule). To address these issues, we proposed the Molecular Classifier Systems (MCS.b), a string-based artificial chemistry based on Holland’s Broadcast Language. In the MCS.b, no explicit fitness function is specified, moreover no distinction is made between messages and rules. In the context of the ESIGNET project, we employ the MCS.b to study a subclass of CAS : Cell Signaling Networks (CSNs) which are complex biochemical networks responsible for coordinating cellular activities. As CSNs occur in cells, these networks must replicate themselves prior to cell division. In this poster we present a series of experiments focusing on the self-replication ability of these CAS

Complex Adaptive Team Systems (CATS): Scaling of a Team Leadership Development Model

Complex adaptive systems (CAS) have been identified as being hard to comprehend, composed of multiple interacting components acting interdependently with overlapping functions aimed at adapting to external/environmental forces. The current theoretical model utilized the natural functions of teams, viewing teams as a complex adaptive system, to develop the structure of the theory of complex adaptive team systems (CATS). The CATS model was formulated around the components of complexity theory (interactions, nonlinearity, interdependency, heterogeneity, complex systems, emergence, self-organizing, and adaptability) to show its utility across multiple domains (the role of leadership, organizational learning, organizational change, collective cognitive structures, innovation, cross-business-unit collaborations). In theorizing the CATS model, a new level of analysis was implemented, the interactions between agents as a move toward emergence in complex systems. The CATS model ultimately provides a model for organizations/institutions to drive knowledge creation and innovation while operating in today’s complexity

Artificial societies and information theory: modelling of sub system formation based on Luhmann's autopoietic theory

This thesis develops a theoretical framework for the generation of artificial societies. In particular it shows how sub-systems emerge when the agents are able to learn and have the ability to communicate. This novel theoretical framework integrates the autopoietic hypothesis of human societies, formulated originally by the German sociologist Luhmann, with concepts of Shannon's information theory applied to adaptive learning agents. Simulations were executed using Multi-Agent-Based Modelling (ABM), a relatively new computational modelling paradigm involving the modelling of phenomena as dynamical systems of interacting agents. The thesis in particular, investigates the functions and properties necessary to reproduce the paradigm of society by using the mentioned ABM approach. Luhmann has proposed that in society subsystems are formed to reduce uncertainty. Subsystems can then be composed by agents with a reduced behavioural complexity. For example in society there are people who produce goods and other who distribute them. Both the behaviour and communication is learned by the agent and not imposed. The simulated task is to collect food, keep it and eat it until sated. Every agent communicates its energy state to the neighbouring agents. This results in two subsystems whereas agents in the first collect food and in the latter steal food from others. The ratio between the number of agents that belongs to the first system and to the second system, depends on the number of food resources. Simulations are in accordance with Luhmann, who suggested that adaptive agents self-organise by reducing the amount of sensory information or, equivalently, reducing the complexity of the perceived environment from the agent's perspective. Shannon's information theorem is used to assess the performance of the simulated learning agents. A practical measure, based on the concept of Shannon's information ow, is developed and applied to adaptive controllers which use Hebbian learning, input correlation learning (ICO/ISO) and temporal difference learning. The behavioural complexity is measured with a novel information measure, called Predictive Performance, which is able to measure at a subjective level how good an agent is performing a task. This is then used to quantify the social division of tasks in a social group of honest, cooperative food foraging, communicating agents

A molecular approach to complex adaptive systems

Complex Adaptive Systems (CAS) are dynamical networks of interacting agents which as a whole determine the behavior, adaptivity and cognitive ability of the system. CAS are ubiquitous and occur in a variety of natural and artificial systems (e.g., cells, societies, stock markets). To study CAS, Holland proposed to employ an agent-based system in which Learning Classifier Systems (LCS) were used to determine the agents behavior and adaptivity. We argue that LCS are limited for the study of CAS: the rule-discovery mechanism is pre-specified and may limit the evolvability of CAS. Secondly, LCS distinguish a demarcation between messages and rules, however operations are reflexive in CAS, e.g., in a cell, an agent (a molecule) may both act as a message (substrate) and as a catalyst (rule). To address these issues, we proposed the Molecular Classifier Systems (MCS.b), a string-based Artificial Chemistry based on Holland’s broadcast language. In the MCS.b, no explicit fitness function or rule discovery mechanism is specified, moreover no distinction is made between messages and rules. In the context of the ESIGNET project, we employ the MCS.b to study a subclass of CAS: Cell Signaling Networks (CSNs) which are complex biochemical networks responsible for coordinating cellular activities. As CSNs occur in cells, these networks must replicate themselves prior to cell division. In this paper we present a series of experiments focusing on the self-replication ability of these CAS. Results indicate counter intuitive outcomes as opposed to those inferred from the literature. This work highlights the current deficit of a theoretical framework for the study of Artificial Chemistries

No-Regret Learning and Equilibrium Computation in Quantum Games

As quantum processors advance, the emergence of large-scale decentralized systems involving interacting quantum-enabled agents is on the horizon. Recent research efforts have explored quantum versions of Nash and correlated equilibria as solution concepts of strategic quantum interactions, but these approaches did not directly connect to decentralized adaptive setups where agents possess limited information. This paper delves into the dynamics of quantum-enabled agents within decentralized systems that employ no-regret algorithms to update their behaviors over time. Specifically, we investigate two-player quantum zero-sum games and polymatrix quantum zero-sum games, showing that no-regret algorithms converge to separable quantum Nash equilibria in time-average. In the case of general multi-player quantum games, our work leads to a novel solution concept, (separable) quantum coarse correlated equilibria (QCCE), as the convergent outcome of the time-averaged behavior no-regret algorithms, offering a natural solution concept for decentralized quantum systems. Finally, we show that computing QCCEs can be formulated as a semidefinite program and establish the existence of entangled (i.e., non-separable) QCCEs, which cannot be approached via the current paradigm of no-regret learning

Analytics and complexity: learning and leading for the future

There is growing interest in the application of learning analytics to manage, inform and improve learning and teaching within higher education. In particular, learning analytics is seen as enabling data-driven decision making as universities are seeking to respond a range of significant challenges that are reshaping the higher education landscape. Experience over four years with a project exploring the use of learning analytics to improve learning and teaching at a particular university has, however, revealed a much more complex reality that potentially limits the value of some analytics-based strategies. This paper uses this experience with over 80,000 students across three learning management systems, combined with literature from complex adaptive systems and learning analytics to identify the source and nature of these limitations along with a suggested path forward

Dynamics of Interacting Neural Networks

The dynamics of interacting perceptrons is solved analytically. For a directed flow of information the system runs into a state which has a higher symmetry than the topology of the model. A symmetry breaking phase transition is found with increasing learning rate. In addition it is shown that a system of interacting perceptrons which is trained on the history of its minority decisions develops a good strategy for the problem of adaptive competition known as the Bar Problem or Minority Game.Comment: 9 pages, 3 figures; typos corrected, content reorganize

Stability and Diversity in Collective Adaptation

We derive a class of macroscopic differential equations that describe collective adaptation, starting from a discrete-time stochastic microscopic model. The behavior of each agent is a dynamic balance between adaptation that locally achieves the best action and memory loss that leads to randomized behavior. We show that, although individual agents interact with their environment and other agents in a purely self-interested way, macroscopic behavior can be interpreted as game dynamics. Application to several familiar, explicit game interactions shows that the adaptation dynamics exhibits a diversity of collective behaviors. The simplicity of the assumptions underlying the macroscopic equations suggests that these behaviors should be expected broadly in collective adaptation. We also analyze the adaptation dynamics from an information-theoretic viewpoint and discuss self-organization induced by information flux between agents, giving a novel view of collective adaptation.Comment: 22 pages, 23 figures; updated references, corrected typos, changed conten