Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learning

Being able to harness the power of large datasets for developing cooperative multi-agent controllers promises to unlock enormous value for real-world applications. Many important industrial systems are multi-agent in nature and are difficult to model using bespoke simulators. However, in industry, distributed processes can often be recorded during operation, and large quantities of demonstrative data stored. Offline multi-agent reinforcement learning (MARL) provides a promising paradigm for building effective decentralised controllers from such datasets. However, offline MARL is still in its infancy and therefore lacks standardised benchmark datasets and baselines typically found in more mature subfields of reinforcement learning (RL). These deficiencies make it difficult for the community to sensibly measure progress. In this work, we aim to fill this gap by releasing off-the-grid MARL (OG-MARL): a growing repository of high-quality datasets with baselines for cooperative offline MARL research. Our datasets provide settings that are characteristic of real-world systems, including complex environment dynamics, heterogeneous agents, non-stationarity, many agents, partial observability, suboptimality, sparse rewards and demonstrated coordination. For each setting, we provide a range of different dataset types (e.g. Good, Medium, Poor, and Replay) and profile the composition of experiences for each dataset. We hope that OG-MARL will serve the community as a reliable source of datasets and help drive progress, while also providing an accessible entry point for researchers new to the field.Comment: Extended Abstract at Autonomous Agents and Multi-Agent Systems Conference 202

An agent-based fuzzy cognitive map approach to the strategic marketing planning for industrial firms

This is the post-print version of the final paper published in Industrial Marketing Management. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2013 Elsevier B.V.Industrial marketing planning is a typical example of an unstructured decision making problem due to the large number of variables to consider and the uncertainty imposed on those variables. Although abundant studies identified barriers and facilitators of effective industrial marketing planning in practice, the literature still lacks practical tools and methods that marketing managers can use for the task. This paper applies fuzzy cognitive maps (FCM) to industrial marketing planning. In particular, agent based inference method is proposed to overcome dynamic relationships, time lags, and reusability issues of FCM evaluation. MACOM simulator also is developed to help marketing managers conduct what-if scenarios to see the impacts of possible changes on the variables defined in an FCM that represents industrial marketing planning problem. The simulator is applied to an industrial marketing planning problem for a global software service company in South Korea. This study has practical implication as it supports marketing managers for industrial marketing planning that has large number of variables and their cause–effect relationships. It also contributes to FCM theory by providing an agent based method for the inference of FCM. Finally, MACOM also provides academics in the industrial marketing management discipline with a tool for developing and pre-verifying a conceptual model based on qualitative knowledge of marketing practitioners.Ministry of Education, Science and Technology (Korea

Evaluating how agent methodologies support the specification of the normative environment through the development process

[EN] Due to the increase in collaborative work and the decentralization of processes in many domains, there is an expanding demand for large-scale, flexible and adaptive software systems to support the interactions of people and institutions distributed in heterogeneous environments. Commonly, these software applications should follow specific regulations meaning the actors using them are bound by rights, duties and restrictions. Since this normative environment determines the final design of the software system, it should be considered as an important issue during the design of the system. Some agent-oriented software engineering methodologies deal with the development of normative systems (systems that have a normative environment) by integrating the analysis of the normative environment of a system in the development process. This paper analyses to what extent these methodologies support the analysis and formalisation of the normative environment and highlights some open issues of the topic.This work is partially supported by the PROMETEOII/2013/019, TIN2012-36586-C03-01, FP7-29493, TIN2011-27652-C03-00, CSD2007-00022 projects, and the CASES project within the 7th European Community Framework Program under the grant agreement No 294931.Garcia Marques, ME.; Miles, S.; Luck, M.; Giret Boggino, AS. (2014). Evaluating how agent methodologies support the specification of the normative environment through the development process. Autonomous Agents and Multi-Agent Systems. 1-20. https://doi.org/10.1007/s10458-014-9275-zS120Cossentino, M., Hilaire, V., Molesini, A., & Seidita, V. (Eds.). (2014). Handbook on agent-oriented design processes (Vol. VIII, 569 p. 508 illus.). Berlin: Springer.Akbari, O. (2010). A survey of agent-oriented software engineering paradigm: Towards its industrial acceptance. Journal of Computer Engineering Research, 1, 14–28.Argente, E., Botti, V., Carrascosa, C., Giret, A., Julian, V., & Rebollo, M. (2011). An abstract architecture for virtual organizations: The THOMAS approach. Knowledge and Information Systems, 29(2), 379–403.Argente, E., Botti, V., & Julian, V. (2009). GORMAS: An organizational-oriented methodological guideline for open MAS. In Proceedings of AOSE’09 (pp. 440–449).Argente, E., Botti, V., & Julian, V. (2009). Organizational-oriented methodological guidelines for designing virtual organizations. In Distributed computing, artificial intelligence, bioinformatics, soft computing, and ambient assisted living. Lecture Notes in Computer Science (Vol. 5518, pp. 154–162).Boella, G., Pigozzi, G., & van der Torre, L. (2009). Normative systems in computer science—Ten guidelines for normative multiagent systems. In G. Boella, P. Noriega, G. Pigozzi, & H. Verhagen (Eds.), Normative multi-agent systems, number 09121 in Dagstuhl seminar proceedings.Boella, G., Torre, L., & Verhagen, H. (2006). Introduction to normative multiagent systems. Computational and Mathematical Organization Theory, 12(2–3), 71–79.Bogdanovych, A., Esteva, M., Simoff, S., Sierra, C., & Berger, H. (2008). A methodology for developing multiagent systems as 3d electronic institutions. In M. Luck & L. Padgham (Eds.), Agent-Oriented Software Engineering VIII (Vol. 4951, pp. 103–117). Lecture Notes in Computer Science. Berlin: Springer.Boissier, O., Padget, J., Dignum, V., Lindemann, G., Matson, E., Ossowski, S., Sichman, J., & Vazquez-Salceda, J. (2006). Coordination, organizations, institutions and norms in multi-agent systems. LNCS (LNAI) (Vol. 3913).Bordini, R. H., Fisher, M., Visser, W., & Wooldridge, M. (2006). Verifying multi-agent programs by model checking. In Autonomous agents and multi-agent systems (Vol. 12, pp. 239–256). Hingham, MA: Kluwer Academic Publishers.Botti, V., Garrido, A., Giret, A., & Noriega, P. (2011). The role of MAS as a decision support tool in a water-rights market. In Post-proceedings workshops AAMAS2011 (Vol. 7068, pp. 35–49). Berlin: Springer.Breaux, T. (2009). Exercising due diligence in legal requirements acquisition: A tool-supported, frame-based approach. In Proceedings of the IEEE international requirements engineering conference (pp. 225–230).Breaux, T. D., & Baumer, D. L. (2011). Legally reasonable security requirements: A 10-year ftc retrospective. Computers and Security, 30(4), 178–193.Breaux, T. D., Vail, M. W., & Anton, A. I. (2006). Towards regulatory compliance: Extracting rights and obligations to align requirements with regulations. In Proceedings of the 14th IEEE international requirements engineering conference, RE ’06 (pp. 46–55). Washington, DC: IEEE Computer Society.Bresciani, P., Perini, A., Giorgini, P., Giunchiglia, F., & Mylopoulos, J. (2004). Tropos: An agent-oriented software development methodology. Autonomous Agents and Multi-Agent Systems, 8(3), 203–236.Cardoso, H. L., & Oliveira, E. (2008). A contract model for electronic institutions. In COIN’07: Proceedings of the 2007 international conference on Coordination, organizations, institutions, and norms in agent systems III (pp. 27–40).Castor, A., Pinto, R. C., Silva, C. T. L. L., & Castro, J. (2004). Towards requirement traceability in tropos. In WER (pp. 189–200).Chopra, A., Dalpiaz, F., Giorgini, P., & Mylopoulos, J. (2009). Modeling and reasoning about service-oriented applications via goals and commitments. ICST conference on digital business.Cliffe, O., Vos, M., & Padget, J. (2006). Specifying and analysing agent-based social institutions using answer set programming. In O. Boissier, J. Padget, V. Dignum, G. Lindemann, E. Matson, S. Ossowski, J. Sichman, & J. Vázquez-Salceda (Eds.), Coordination, organizations, institutions, and norms in multi-agent systems. Lecture Notes in Computer Science (Vol. 3913, pp. 99–113). Springer. Berlin.Criado, N., Argente, E., Garrido, A., Gimeno, J. A., Igual, F., Botti, V., Noriega, P., & Giret, A. (2011). Norm enforceability in Electronic Institutions? In Coordination, organizations, institutions, and norms in agent systems VI (Vol. 6541, pp. 250–267). Springer.Dellarocas, C., & Klein, M. (2001). Contractual agent societies. In R. Conte & C. Dellarocas (Eds.), Social order in multiagent systems (Vol. 2, pp. 113–133)., Multiagent Systems, Artificial Societies, and Simulated Organizations New York: Springer.DeLoach, S. A. (2008). Developing a multiagent conference management system using the o-mase process framework. In Proceedings of the international conference on agent-oriented software engineering VIII (pp. 168–181).DeLoach, S. A., & Garcia-Ojeda, J. C. (2010). O-mase; a customisable approach to designing and building complex, adaptive multi-agent systems. International Journal of Agent-Oriented Software Engineering, 4(3), 244–280.DeLoach, S. A., Padgham, L., Perini, A., Susi, A., & Thangarajah, J. (2009). Using three aose toolkits to develop a sample design. International Journal Agent-Oriented Software Engineering, 3, 416–476.Dignum, F., Dignum, V., Thangarajah, J., Padgham, L., & Winikoff, M. (2007). Open agent systems? Eighth international workshop on agent oriented software engineering (AOSE) in AAMAS07.Dignum, V. (2003). A model for organizational interaction:based on agents, founded in logic. PhD thesis, Utrecht University.Dignum, V., Meyer, J., Dignum, F., & Weigand, H. (2003). Formal specification of interaction in agent societies. Formal approaches to agent-based systems (Vol. 2699).Dignum, V., Vazquez-Salceda, J., & Dignum, F. (2005). Omni: Introducing social structure, norms and ontologies into agent organizations. In R. Bordini, M. Dastani, J. Dix, & A. Seghrouchni (Eds.)Programming multi-agent systems. Lecture Notes in Computer Science (Vol. 3346, pp. 181–198). Berlin: Springer.d’Inverno, M., Luck, M., Noriega, P., Rodriguez-Aguilar, J., & Sierra, C. (2012). Communicating open systems, 186, 38–94.Elsenbroich, C., & Gilbert, N. (2014). Agent-based modelling. In Modelling norms (pp. 65–84). Dordrecht: Springer.Esteva, M., Rosell, B., Rodriguez, J. A., & Arcos, J. L. (2004). AMELI: An agent-based middleware for electronic institutions. In AAMAS04 (pp. 236–243).Fenech, S., Pace, G. J., & Schneider, G. (2009). Automatic conflict detection on contracts. In Proceedings of the 6th international colloquium on theoretical aspects of computing, ICTAC ’09 (pp. 200–214).Garbay, C., Badeig, F., & Caelen, J. (2012). Normative multi-agent approach to support collaborative work in distributed tangible environments. In Proceedings of the ACM 2012 conference on computer supported cooperative work companion, CSCW ’12 (pp. 83–86). New York, NY: ACM.Garcia, E., Giret, A., & Botti, V. (2011). Regulated open multi-agent systems based on contracts. In Information Systems Development (pp. 243–255).Garcia, E., Tyson, G., Miles, S., Luck, M., Taweel, A., Staa, T. V., & Delaney, B. (2012). An analysis of agent-oriented engineering of e-health systems. In 13th international eorkshop on sgent-oriented software engineering (AOSE-AAMAS).Garcia, E., Tyson, G., Miles, S., Luck, M., Taweel, A., Staa, T. V., and Delaney, B. (2013). Analysing the Suitability of Multiagent Methodologies for e-Health Systems. In Agent-Oriented Software Engineering XIII, volume 7852, pages 134–150. Springer-Verlag.Garrido, A., Giret, A., Botti, V., & Noriega, P. (2013). mWater, a case study for modeling virtual markets. In New perspectives on agreement technologies (Vol. Law, Gover, pp. 563–579). Springer.Gteau, B., Boissier, O., & Khadraoui, D. (2006). Multi-agent-based support for electronic contracting in virtual enterprises. IFAC Symposium on Information Control Problems in Manufacturing (INCOM), 150(3), 73–91.Hollander, C. D., & Wu, A. S. (2011). The current state of normative agent-based systems. Journal of Artificial Societies and Social Simulation, 14(2), 6.Hsieh, F.-S. (2005). Automated negotiation based on contract net and petri net. In E-commerce and web technologies. Lecture Notes in Computer Science (Vol. 3590, pp. 148–157).Kollingbaum, M., Jureta, I. J., Vasconcelos, W., & Sycara, K. (2008). Automated requirements-driven definition of norms for the regulation of behavior in multi-agent systems. In Proceedings of the AISB 2008 workshop on behaviour regulation in multi-agent systems, Aberdeen, Scotland, U.K., April 2008.Li, T., Balke, T., Vos, M., Satoh, K., & Padget, J. (2013). Detecting conflicts in legal systems. In Y. Motomura, A. Butler, & D. Bekki (Eds.), New Frontiers in Artificial Intelligence (Vol. 7856, pp. 174–189)., Lecture Notes in Computer Science Berlin Heidelberg: Springer.Lomuscio, A., Qu, H., & Solanki, M. (2010) Towards verifying contract regulated service composition. Journal of Autonomous Agents and Multi-Agent Systems (pp. 1–29).Lopez, F., Luck, M., & d’Inverno, M. (2006). A normative framework for agent-based systems. Computational and Mathematical Organization Theory, 12, 227–250.Lpez, F. y, Luck, M., & dInverno, M. (2006). A normative framework for agent-based systems. Computational and Mathematical Organization Theory, 12(2–3), 227–250.Mader, P., & Egyed, A. (2012). Assessing the effect of requirements traceability for software maintenance. In 28th IEEE International Conference on Software Maintenance (ICSM) (pp. 171–180), Sept 2012.Mao, X., & Yu, E. (2005). Organizational and social concepts in agent oriented software engineering. In AOSE IV. Lecture Notes in Artificial Intelligence (Vol. 3382, pp. 184–202).Meyer, J.-J. C., & Wieringa, R. J. (Eds.). (1993). Deontic logic in computer science: Normative system specification. Chichester, UK: Wiley.Okouya, D., & Dignum, V. (2008). Operetta: A prototype tool for the design, analysis and development of multi-agent organizations (demo paper). In AAMAS (pp. 1667–1678).Malone, T. W., Smith J. B., & Olson, G. M. (2001). Coordination theory and collaboration technology. Mahwah, NJ: Lawrence Erlbaum Associates.Oren, N., Panagiotidi, S., Vázquez-Salceda, J., Modgil, S., Luck, M., & Miles, S. (2009). Towards a formalisation of electronic contracting environments. COIN (pp. 156–171).Osman, N., Robertson, D., & Walton, C. (2006). Run-time model checking of interaction and deontic models for multi-agent systems. In AAMAS ’06: Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems (pp. 238–240). New York, NY: ACM.Pace, G., Prisacariu, C., & Schneider, G. (2007). Model checking contracts a case study. In Automated technology for verification and analysis. Lecture Notes in Computer Science (Vol. 4762, pp. 82–97).Rotolo, A., & van der Torre, L. (2011). Rules, agents and norms: Guidelines for rule-based normative multi-agent systems. RuleML Europe, 6826, 52–66.Saeki, M., & Kaiya, H. (2008). Supporting the elicitation of requirements compliant with regulations. In CAiSE ’08 (pp. 228–242).Siena, A., Mylopoulos, J., Perini, A., & Susi, A. (2009). Designing law-compliant software requirements. In Proceedings of the 28th international conference on conceptual modeling, ER ’09 (pp. 472–486).Singh, M. P. Commitments in multiagent systems: Some history, some confusions, some controversies, some prospects.Solaiman, E., Molina-Jimenez, C., & Shrivastav, S. (2003). Model checking correctness properties of electronic contracts. In Service-oriented computing—ICSOC 2003. Lecture Notes in Computer Science (Vol. 2910, pp. 303–318). Berlin: Springer.Telang, P. R., & Singh, M. P. (2009). Conceptual modeling: Foundations and applications. Enhancing tropos with commitments (pp. 417–435).Vázquez-Salceda, J., Confalonieri, R., Gomez, I., Storms, P., Nick Kuijpers, S. P., & Alvarez, S. (2009). Modelling contractually-bounded interactions in the car insurance domain. DIGIBIZ 2009.Viganò, F., & Colombetti, M. (2007). Symbolic model checking of institutions. In ICEC (pp. 35–44).Walton, C. D. (2007). Verifiable agent dialogues. Journal of Applied Logic, 5(2):197–213, Logic-Based Agent Verification.Winkler, S., & Pilgrim, J. (2010). A survey of traceability in requirements engineering and model-driven development. Software and Systems Modeling (SoSyM), 9(4), 529–565.Wooldridge, M., Fisher, M., Huget, M., & Parsons, S. (2002). Model checking multi-agent systems with mable. In AAMAS02 (pp. 952–959). ACM

Industrial Symbiotic Networks as Coordinated Games

We present an approach for implementing a specific form of collaborative industrial practices-called Industrial Symbiotic Networks (ISNs)-as MC-Net cooperative games and address the so called ISN implementation problem. This is, the characteristics of ISNs may lead to inapplicability of fair and stable benefit allocation methods even if the collaboration is a collectively desired one. Inspired by realistic ISN scenarios and the literature on normative multi-agent systems, we consider regulations and normative socioeconomic policies as two elements that in combination with ISN games resolve the situation and result in the concept of coordinated ISNs.Comment: 3 pages, Proc. of the 17th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2018

Development of a Multi-agent Collision Resolution System at the Supply of Spare Parts and Components to the Production Equipment of Industrial Enterprises

The approach to the creation of computer facilities for the automation of the technical maintenance of production equipment (TMPE) at industrial enterprises (IE) is outlined. Meaningful and formal statement of the problem of forming solutions for identifying and eliminating collisions that arise when delivering spare parts and components for TMPE are presented. The method of formation of coordinating decisions on maintenance with spare parts and accessories for carrying out TMPE at IE is described. The organization of intellectual support of formation of coordinating decisions by recognition of potential collision in the TMPE process is offered. This procedure involves checking the real existence of the collision and issuing a coordinating decision. In this case, the decision is formed in the event of a disagreement between the need for spare parts and components for the TMPE maintenance, with their availability in the PP warehouse. The ways of software implementation of this method in the environment of multi-agent system are considered. In particular, the description of the multi-agent system developed during the prototype research is given. The prototype is implemented using CORBA technology, in accordance with DSTU ISO/ EC 2382-15:2005. The calculation of the efficiency of the application of the developed computer tools in production is shown. To assess the quality of the system, a sliding control method based on leave-on-out cross-validation (LOOCV) is applied

An Agent-based approach to modelling integrated product teams undertaking a design activity.

The interactions between individual designers, within integrated product teams, and the nature of design tasks, all have a significant impact upon how well a design task can be performed, and hence the quality of the resultant product and the time in which it can be delivered. In this paper we describe an ongoing research project which aims to model integrated product teams through the use of multi-agent systems. We first describe the background and rationale for our work, and then present our initial computational model and results from the simulation of an integrated product team. The paper concludes with a discussion of how the model will evolve to improve the accuracy of the simulation

Blockchain Solutions for Multi-Agent Robotic Systems: Related Work and Open Questions

The possibilities of decentralization and immutability make blockchain probably one of the most breakthrough and promising technological innovations in recent years. This paper presents an overview, analysis, and classification of possible blockchain solutions for practical tasks facing multi-agent robotic systems. The paper discusses blockchain-based applications that demonstrate how distributed ledger can be used to extend the existing number of research platforms and libraries for multi-agent robotic systems.Comment: 5 pages, FRUCT-2019 conference pape