Aprendizagem de coordenação em sistemas multi-agente

The ability for an agent to coordinate with others within a system is a valuable property in multi-agent systems. Agents either cooperate as a team to accomplish a common goal, or adapt to opponents to complete different goals without being exploited. Research has shown that learning multi-agent coordination is significantly more complex than learning policies in singleagent environments, and requires a variety of techniques to deal with the properties of a system where agents learn concurrently. This thesis aims to determine how can machine learning be used to achieve coordination within a multi-agent system. It asks what techniques can be used to tackle the increased complexity of such systems and their credit assignment challenges, how to achieve coordination, and how to use communication to improve the behavior of a team. Many algorithms for competitive environments are tabular-based, preventing their use with high-dimension or continuous state-spaces, and may be biased against specific equilibrium strategies. This thesis proposes multiple deep learning extensions for competitive environments, allowing algorithms to reach equilibrium strategies in complex and partially-observable environments, relying only on local information. A tabular algorithm is also extended with a new update rule that eliminates its bias against deterministic strategies. Current state-of-the-art approaches for cooperative environments rely on deep learning to handle the environment’s complexity and benefit from a centralized learning phase. Solutions that incorporate communication between agents often prevent agents from being executed in a distributed manner. This thesis proposes a multi-agent algorithm where agents learn communication protocols to compensate for local partial-observability, and remain independently executed. A centralized learning phase can incorporate additional environment information to increase the robustness and speed with which a team converges to successful policies. The algorithm outperforms current state-of-the-art approaches in a wide variety of multi-agent environments. A permutation invariant network architecture is also proposed to increase the scalability of the algorithm to large team sizes. Further research is needed to identify how can the techniques proposed in this thesis, for cooperative and competitive environments, be used in unison for mixed environments, and whether they are adequate for general artificial intelligence.A capacidade de um agente se coordenar com outros num sistema é uma propriedade valiosa em sistemas multi-agente. Agentes cooperam como uma equipa para cumprir um objetivo comum, ou adaptam-se aos oponentes de forma a completar objetivos egoístas sem serem explorados. Investigação demonstra que aprender coordenação multi-agente é significativamente mais complexo que aprender estratégias em ambientes com um único agente, e requer uma variedade de técnicas para lidar com um ambiente onde agentes aprendem simultaneamente. Esta tese procura determinar como aprendizagem automática pode ser usada para encontrar coordenação em sistemas multi-agente. O documento questiona que técnicas podem ser usadas para enfrentar a superior complexidade destes sistemas e o seu desafio de atribuição de crédito, como aprender coordenação, e como usar comunicação para melhorar o comportamento duma equipa. Múltiplos algoritmos para ambientes competitivos são tabulares, o que impede o seu uso com espaços de estado de alta-dimensão ou contínuos, e podem ter tendências contra estratégias de equilíbrio específicas. Esta tese propõe múltiplas extensões de aprendizagem profunda para ambientes competitivos, permitindo a algoritmos atingir estratégias de equilíbrio em ambientes complexos e parcialmente-observáveis, com base em apenas informação local. Um algoritmo tabular é também extendido com um novo critério de atualização que elimina a sua tendência contra estratégias determinísticas. Atuais soluções de estado-da-arte para ambientes cooperativos têm base em aprendizagem profunda para lidar com a complexidade do ambiente, e beneficiam duma fase de aprendizagem centralizada. Soluções que incorporam comunicação entre agentes frequentemente impedem os próprios de ser executados de forma distribuída. Esta tese propõe um algoritmo multi-agente onde os agentes aprendem protocolos de comunicação para compensarem por observabilidade parcial local, e continuam a ser executados de forma distribuída. Uma fase de aprendizagem centralizada pode incorporar informação adicional sobre ambiente para aumentar a robustez e velocidade com que uma equipa converge para estratégias bem-sucedidas. O algoritmo ultrapassa abordagens estado-da-arte atuais numa grande variedade de ambientes multi-agente. Uma arquitetura de rede invariante a permutações é também proposta para aumentar a escalabilidade do algoritmo para grandes equipas. Mais pesquisa é necessária para identificar como as técnicas propostas nesta tese, para ambientes cooperativos e competitivos, podem ser usadas em conjunto para ambientes mistos, e averiguar se são adequadas a inteligência artificial geral.Apoio financeiro da FCT e do FSE no âmbito do III Quadro Comunitário de ApoioPrograma Doutoral em Informátic

Collective Information Processing and Criticality, Evolution and Limited Attention.

Im ersten Teil analysiere ich die Selbstorganisation zur Kritikalität (hier ein Phasenübergang von Ordnung zu Unordnung) und untersuche, ob Evolution ein möglicher Organisationsmechanismus ist. Die Kernfrage ist, ob sich ein simulierter kohäsiver Schwarm, der versucht, einem Raubtier auszuweichen, durch Evolution selbst zum kritischen Punkt entwickelt, um das Ausweichen zu optimieren? Es stellt sich heraus, dass (i) die Gruppe den Jäger am besten am kritischen Punkt vermeidet, aber (ii) nicht durch einer verstärkten Reaktion, sondern durch strukturelle Veränderungen, (iii) das Gruppenoptimum ist evolutionär unstabiler aufgrund einer maximalen räumlichen Selbstsortierung der Individuen. Im zweiten Teil modelliere ich experimentell beobachtete Unterschiede im kollektiven Verhalten von Fischgruppen, die über mehrere Generationen verschiedenen Arten von größenabhängiger Selektion ausgesetzt waren. Diese Größenselektion soll Freizeitfischerei (kleine Fische werden freigelassen, große werden konsumiert) und die kommerzielle Fischerei mit großen Netzbreiten (kleine/junge Individuen können entkommen) nachahmen. Die zeigt sich, dass das Fangen großer Fische den Zusammenhalt und die Risikobereitschaft der Individuen reduziert. Beide Befunde lassen sich mechanistisch durch einen Aufmerksamkeits-Kompromiss zwischen Sozial- und Umweltinformationen erklären. Im letzten Teil der Arbeit quantifiziere ich die kollektive Informationsverarbeitung im Feld. Das Studiensystem ist eine an sulfidische Wasserbedingungen angepasste Fischart mit einem kollektiven Fluchtverhalten vor Vögeln (wiederholte kollektive Fluchttauchgängen). Die Fische sind etwa 2 Zentimeter groß, aber die kollektive Welle breitet sich über Meter in dichten Schwärmen an der Oberfläche aus. Es zeigt sich, dass die Wellengeschwindigkeit schwach mit der Polarisation zunimmt, bei einer optimalen Dichte am schnellsten ist und von ihrer Richtung relativ zur Schwarmorientierung abhängt.In the first part, I focus on the self-organization to criticality (here an order-disorder phase transition) and investigate if evolution is a possible self-tuning mechanism. Does a simulated cohesive swarm that tries to avoid a pursuing predator self-tunes itself by evolution to the critical point to optimize avoidance? It turns out that (i) the best group avoidance is at criticality but (ii) not due to an enhanced response but because of structural changes (fundamentally linked to criticality), (iii) the group optimum is not an evolutionary stable state, in fact (iv) it is an evolutionary accelerator due to a maximal spatial self-sorting of individuals causing spatial selection. In the second part, I model experimentally observed differences in collective behavior of fish groups subject to multiple generation of different types of size-dependent selection. The real world analog to this experimental evolution is recreational fishery (small fish are released, large are consumed) and commercial fishing with large net widths (small/young individuals can escape). The results suggest that large harvesting reduces cohesion and risk taking of individuals. I show that both findings can be mechanistically explained based on an attention trade-off between social and environmental information. Furthermore, I numerically analyze how differently size-harvested groups perform in a natural predator and fishing scenario. In the last part of the thesis, I quantify the collective information processing in the field. The study system is a fish species adapted to sulfidic water conditions with a collective escape behavior from aerial predators which manifests in repeated collective escape dives. These fish measure about 2 centimeters, but the collective wave spreads across meters in dense shoals at the surface. I find that wave speed increases weakly with polarization, is fastest at an optimal density and depends on its direction relative to shoal orientation

採餌問題のための確率的探索戦略の設計と最適化

Autonomous robot’s search strategy is the set of rules that it employs while looking for targets in its environment. In this study, the stochastic movement of robots in unknown environments is statistically studied, using a Levy walk method. Biological systems (e.g., foraging animals) provide useful models for designing optimal stochastic search algorithms. Observations of biological systems, ranging from large animals to immune cells, have inspired the design of efficient search strategies that incorporate stochastic movement. In this study, we seek to identify the optimal stochastic strategies for autonomous robots. Given the complexity of interaction between the robot and its environment, optimization must be performed in high-dimensional parameter space. The effect of the explanatory variable on the forger robot movement with the minimum required energy was also studied using experiments done by the response surface methodology (RSM). We analyzed the extent to which search efficiency requires these characteristics, using RSM. Correlation between the involved parameters via a Lévy walk process was examined through designing a setup for the experiments to determine the interaction of the involved variables and the robot movement. The extracted statistical model represents the priority influence of those variables on the robot by developing the statistical model of the mentioned unknown area. The efficiency of a simple strategy was investigated based on Lévy walk search in two-dimensional landscapes with clumped resource distributions. We show how RSM techniques can be used to identify optimal parameter values as well as to describe how sensitive efficiency reacts to the changes in these values. Here, we identified optimal parameter for designing robot by using stochastic search pattern and applying mood-switching criteria on a mixture of speed and sensor and μ to determine how many robots are needed for a solution. Fractal criterion-based robot strategies were more efficient than those based on the resource encounter criterion, and the former was found to be more robust to changes in resource distribution as well.九州工業大学博士学位論文 学位記番号:生工博甲第358号 学位授与年月日:令和元年9月20日1 Introduction|2 Levy Walk|3 Design of Experiment (DOE)|4 Response Surface Methodology|5 Results and Discussions|6 Conclusion九州工業大学令和元年

Art Informing Science Education: The Potential Contributions of Ornithological Illustration to Ecology Education

Birds serve as an excellent group of organisms from which to introduce the study of ecology, being of inherent aesthetic interest to many otherwise uninterested in science, and are also ubiquitous in the immediate environment of many students. By extension, images of birds might serve as a valuable resource for the ecology educator, and bird artists – as a subset of ecologists - might provide useful models for expertise in ecology. This study examines the potential contributions of bird artists and bird art to education in ecology at the high school and college level. Eight contemporary bird artists were interviewed in depth to provide a multiple case study for the development of expertise in field ornithology as a subspecialty of ecology. Forty narrative bird paintings and forty sets of plates from popular field guides to birds were analyzed for their potential classroom use. Twenty-five ecological concepts were identified within the sample of bird art examined, all of which are recommended for inclusion in the ecology curriculum. Additionally, images of birds were found to have considerable potential for illustrating and teaching the history and nature of science

Aprendizagem a partir de múltiplas fontes em grupos heterogéneos de agentes

Tese de doutoramento. Engenharia Informática. Faculdade de Engenharia. Universidade do Porto. 200

Online child sexual exploitation: a new MIS challenge

© 2021, Association for Information Systems. All rights reserved. This paper deals with the difficult yet increasingly important MIS phenomenon of online child sexual exploitation (online CSE). Through the use of secondary and publicly available data from the Federal Bureau of Investigation, as well as primary data from a cybercrime police unit in the United Kingdom, this study takes a grounded theory approach and organizes the role that technologies and social actors play in shaping online CSE. The paper contributes to IS theory by providing a consolidated model for online CSE, which we call the technology and imagery dimensions model. This model combines the staging of the phenomenon and the key dimensions that depict how the use of technology and imagery both fuels and defuses the phenomenon. In informing the construction of the model, the paper extracts, organizes, and generalizes the affordances of technology and discusses the role of information systems in detecting online CSE

The Influence of Collective Working Memory Strategies on Agent Teams

Past self-organizing models of collectively moving "particles" (simulated bird flocks, fish schools, etc.) typically have been based on purely reflexive agents that have no significant memory of past movements or environmental obstacles. These agent collectives usually operate in abstract environments, but as these domains take on a greater realism, the collective requires behaviors use not only presently observed stimuli but also remembered information. It is hypothesized that the addition of a limited working memory of the environment, distributed among the collective's individuals can improve efficiency in performing tasks. This is first approached in a more traditional particle system in an abstract environment. Then it is explored for a single agent, and finally a team of agents, operating in a simulated 3-dimensional environment of greater realism. In the abstract environment, a limited distributed working memory produced a significant improvement in travel between locations, in some cases improving performance over time, while in others surprisingly achieving an immediate benefit from the influence of memory. When strategies for accumulating and manipulating memory were subsequently explored for a more realistic single agent in the 3-dimensional environment, if the agent kept a local or a cumulative working memory, its performance improved on different tasks, both when navigating nearby obstacles and, in the case of cumulative memory, when covering previously traversed terrain. When investigating a team of these agents engaged in a pursuit scenario, it was determined that a communicating and coordinating team still benefited from a working memory of the environment distributed among the agents, even with limited memory capacity. This demonstrates that a limited distributed working memory in a multi-agent system improves performance on tasks in domains of increasing complexity. This is true even though individual agents know only a fraction of the collective's entire memory, using this partial memory and interactions with others in the team to perform tasks. These results may prove useful in improving existing methodologies for control of collective movements for robotic teams, computer graphics, particle swarm optimization, and computer games, and in interpreting future experimental research on group movements in biological populations

Using MapReduce Streaming for Distributed Life Simulation on the Cloud

Distributed software simulations are indispensable in the study of large-scale life models but often require the use of technically complex lower-level distributed computing frameworks, such as MPI. We propose to overcome the complexity challenge by applying the emerging MapReduce (MR) model to distributed life simulations and by running such simulations on the cloud. Technically, we design optimized MR streaming algorithms for discrete and continuous versions of Conway’s life according to a general MR streaming pattern. We chose life because it is simple enough as a testbed for MR’s applicability to a-life simulations and general enough to make our results applicable to various lattice-based a-life models. We implement and empirically evaluate our algorithms’ performance on Amazon’s Elastic MR cloud. Our experiments demonstrate that a single MR optimization technique called strip partitioning can reduce the execution time of continuous life simulations by 64%. To the best of our knowledge, we are the first to propose and evaluate MR streaming algorithms for lattice-based simulations. Our algorithms can serve as prototypes in the development of novel MR simulation algorithms for large-scale lattice-based a-life models.https://digitalcommons.chapman.edu/scs_books/1014/thumbnail.jp

Proceedings of The Multi-Agent Logics, Languages, and Organisations Federated Workshops (MALLOW 2010)

http://ceur-ws.org/Vol-627/allproceedings.pdfInternational audienceMALLOW-2010 is a third edition of a series initiated in 2007 in Durham, and pursued in 2009 in Turin. The objective, as initially stated, is to "provide a venue where: the cost of participation was minimum; participants were able to attend various workshops, so fostering collaboration and cross-fertilization; there was a friendly atmosphere and plenty of time for networking, by maximizing the time participants spent together"

Cooperation and Social Dilemmas with Reinforcement Learning

Cooperation between humans has been foundational for the development of civilisation and yet there are many questions about how it emerges from social interactions. As artificial agents begin to play a more significant role in our lives and are introduced into our societies, it is apparent that understanding the mechanisms of cooperation is important also for the design of next-generation multi-agent AI systems. Indeed, this is particularly important in the case of supporting cooperation between self-interested AI agents. In this thesis, we focus on the analysis of the application of mechanisms that are at the basis of human cooperation to the training of reinforcement learning agents. Human behaviour is a product of cultural norms, emotions and intuition amongst other things: we argue it is possible to use similar mechanisms to deal with the complexities of multi-agent cooperation. We outline the problem of cooperation in mixed-motive games, also known as social dilemmas, and we focus on the mechanisms of reputation dynamics and partner selection, two mechanisms that have been strongly linked to indirect reciprocity in Evolutionary Game Theory. A key point that we want to emphasise is the fact we assume no prior knowledge and explicit definition of strategies, which instead are fully learnt by the agents during the games. In our experimental evaluation, we demonstrate the benefits of applying these mechanisms to the training process of the agents, and we compare our findings with results presented in a variety of other disciplines, including Economics and Evolutionary Biology