Evaluating Multi-Agent Coordination Abilities in Large Language Models

A pivotal aim in contemporary AI research is to develop agents proficient in multi-agent coordination, enabling effective collaboration with both humans and other systems. Large Language Models (LLMs), with their notable ability to understand, generate, and interpret language in a human-like manner, stand out as promising candidates for the development of such agents. In this study, we build and assess the effectiveness of agents crafted using LLMs in various coordination scenarios. We introduce the LLM-Coordination (LLM-Co) Framework, specifically designed to enable LLMs to play coordination games. With the LLM-Co framework, we conduct our evaluation with three game environments and organize the evaluation into five aspects: Theory of Mind, Situated Reasoning, Sustained Coordination, Robustness to Partners, and Explicit Assistance. First, the evaluation of the Theory of Mind and Situated Reasoning reveals the capabilities of LLM to infer the partner's intention and reason actions accordingly. Then, the evaluation around Sustained Coordination and Robustness to Partners further showcases the ability of LLMs to coordinate with an unknown partner in complex long-horizon tasks, outperforming Reinforcement Learning baselines. Lastly, to test Explicit Assistance, which refers to the ability of an agent to offer help proactively, we introduce two novel layouts into the Overcooked-AI benchmark, examining if agents can prioritize helping their partners, sacrificing time that could have been spent on their tasks. This research underscores the promising capabilities of LLMs in sophisticated coordination environments and reveals the potential of LLMs in building strong real-world agents for multi-agent coordination

State of the art of a multi-agent based recommender system for active software engineering ontology

Software engineering ontology was first developed to provide efficient collaboration and coordination among distributed teams working on related software development projects across the sites. It helped to clarify the software engineering concepts and project information as well as enable knowledge sharing. However, a major challenge of the software engineering ontology users is that they need the competence to access and translate what they are looking for into the concepts and relations described in the ontology; otherwise, they may not be able to obtain required information. In this paper, we propose a conceptual framework of a multi-agent based recommender system to provide active support to access and utilize knowledge and project information in the software engineering ontology. Multi-agent system and semantic-based recommendation approach will be integrated to create collaborative working environment to access and manipulate data from the ontology and perform reasoning as well as generate expert recommendation facilities for dispersed software teams across the sites

Separating Agent-Functioning and Inter-Agent Coordination by Activated Modules: The DECOMAS Architecture

The embedding of self-organizing inter-agent processes in distributed software applications enables the decentralized coordination system elements, solely based on concerted, localized interactions. The separation and encapsulation of the activities that are conceptually related to the coordination, is a crucial concern for systematic development practices in order to prepare the reuse and systematic integration of coordination processes in software systems. Here, we discuss a programming model that is based on the externalization of processes prescriptions and their embedding in Multi-Agent Systems (MAS). One fundamental design concern for a corresponding execution middleware is the minimal-invasive augmentation of the activities that affect coordination. This design challenge is approached by the activation of agent modules. Modules are converted to software elements that reason about and modify their host agent. We discuss and formalize this extension within the context of a generic coordination architecture and exemplify the proposed programming model with the decentralized management of (web) service infrastructures

Run-Time Selection of Coordination Mechanisms in Multi-Agent Systems

This paper presents a framework that enables autonomous agents to dynamically select the mechanism they employ in order to coordinate their inter-related activities. Adopting this framework means coordination mechanisms move from the realm of being imposed upon the system at design time, to something that the agents select at run-time in order to fit their prevailing circumstances and their current coordination needs. Empirical analysis is used to evaluate the effect of various design alternatives for the agent's decision making mechanisms and for the coordination mechanisms themselves

Coordination approaches and systems - part I : a strategic perspective

This is the first part of a two-part paper presenting a fundamental review and summary of research of design coordination and cooperation technologies. The theme of this review is aimed at the research conducted within the decision management aspect of design coordination. The focus is therefore on the strategies involved in making decisions and how these strategies are used to satisfy design requirements. The paper reviews research within collaborative and coordinated design, project and workflow management, and, task and organization models. The research reviewed has attempted to identify fundamental coordination mechanisms from different domains, however it is concluded that domain independent mechanisms need to be augmented with domain specific mechanisms to facilitate coordination. Part II is a review of design coordination from an operational perspective

Analysis and design of multiagent systems using MAS-CommonKADS

This article proposes an agent-oriented methodology called MAS-CommonKADS and develops a case study. This methodology extends the knowledge engineering methodology CommonKADSwith techniquesfrom objectoriented and protocol engineering methodologies. The methodology consists of the development of seven models: Agent Model, that describes the characteristics of each agent; Task Model, that describes the tasks that the agents carry out; Expertise Model, that describes the knowledge needed by the agents to achieve their goals; Organisation Model, that describes the structural relationships between agents (software agents and/or human agents); Coordination Model, that describes the dynamic relationships between software agents; Communication Model, that describes the dynamic relationships between human agents and their respective personal assistant software agents; and Design Model, that refines the previous models and determines the most suitable agent architecture for each agent, and the requirements of the agent network

Capturing the behaviour of inter-agent dialogues

A multiagent system (MAS) is made up of multiple interacting autonomous agents. It can be viewed as a society in which each agent performs its activity, cooperating to achieve common goals, or competing for them. Thus, every agent has the ability to do social interactions with other agents establishing dialogues via some kind of agent-communication language, under some communication protocol [13]. Argumentation has been used to model several kind of dialogues in multi-agents systems, such as negotiation or coordination [1, 7, 8, 5, 9]. Our current research activities are related to the use of argumentation in agent’s interaction, as a form of social dialogue. According to [15], dialogues can be classified in negotiation, where there is a conflict of interests, persuasion where there is a conflict of opinion or beliefs, indagation where there is a need for an explanation or proof of some proposition, deliberation or coordination where there is a need to coordinate goals and actions, and one special kind of dialogue called eristic based on personal conflicts. Except the last one, all this dialogues may exist in multi-agents systems as part of social activities among agents. We also study the use of argumentation formalisms to model the internal process of reasoning of an agent, often called monologues. Our aim is to define an abstract argumentation framework to capture the behaviour of these different dialogues. We are not interested in the logic used to construct arguments. Our formulation completely abstracts from the internal structure of the arguments, considering them as moves made in a dialogue. We also consider multiagent systems as a set of multiple interacting autonomous agents.Eje: Inteligencia artificial distribuida, aspectos teóricos de la inteligencia artificial y teoría de computaciónRed de Universidades con Carreras en Informática (RedUNCI