Agent Environments for Multi-agent Systems – A Research Roadmap

Ten years ago, researchers in multi-agent systems became more and more aware that agent systems consist of more than only agents. The series of workshops on Environments for Multi-Agent Systems (E4MAS 2004-2006) emerged from this awareness. One of the primary outcomes of this endeavor was a principled understanding that the agent environment should be considered as a primary design abstraction, equally important as the agents. A special issue in JAAMAS 2007 contributed a set of influential papers that define the role of agent environments, describe their engineering, and outline challenges in the field that have been the drivers for numerous follow up research efforts. The goal of this paper is to wrap up what has been achieved in the past 10 years and identify challenges for future research on agent environments. Instead of taking a broad perspective, we focus on three particularly relevant topics of modern software intensive systems: large scale, openness, and humans in the loop. For each topic, we reflect on the challenges outlined 10 years ago, present an example application that highlights the current trends, and from that outline challenges for the future. We conclude with a roadmap on how the different challenges could be tackled. © Springer International Publishing Switzerland 2015.Peer reviewe

PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planning

We present PRM-RL, a hierarchical method for long-range navigation task completion that combines sampling based path planning with reinforcement learning (RL). The RL agents learn short-range, point-to-point navigation policies that capture robot dynamics and task constraints without knowledge of the large-scale topology. Next, the sampling-based planners provide roadmaps which connect robot configurations that can be successfully navigated by the RL agent. The same RL agents are used to control the robot under the direction of the planning, enabling long-range navigation. We use the Probabilistic Roadmaps (PRMs) for the sampling-based planner. The RL agents are constructed using feature-based and deep neural net policies in continuous state and action spaces. We evaluate PRM-RL, both in simulation and on-robot, on two navigation tasks with non-trivial robot dynamics: end-to-end differential drive indoor navigation in office environments, and aerial cargo delivery in urban environments with load displacement constraints. Our results show improvement in task completion over both RL agents on their own and traditional sampling-based planners. In the indoor navigation task, PRM-RL successfully completes up to 215 m long trajectories under noisy sensor conditions, and the aerial cargo delivery completes flights over 1000 m without violating the task constraints in an environment 63 million times larger than used in training.Comment: 9 pages, 7 figure

Challenges for adaptation in agent societies

The final publication is available at Springer via http://dx.doi.org/[insert DOIAdaptation in multiagent systems societies provides a paradigm for allowing these societies to change dynamically in order to satisfy the current requirements of the system. This support is especially required for the next generation of systems that focus on open, dynamic, and adaptive applications. In this paper, we analyze the current state of the art regarding approaches that tackle the adaptation issue in these agent societies. We survey the most relevant works up to now in order to highlight the most remarkable features according to what they support and how this support is provided. In order to compare these approaches, we also identify different characteristics of the adaptation process that are grouped in different phases. Finally, we discuss some of the most important considerations about the analyzed approaches, and we provide some interesting guidelines as open issues that should be required in future developments.This work has been partially supported by CONSOLIDER-INGENIO 2010 under grant CSD2007-00022, the European Cooperation in the field of Scientific and Technical Research IC0801 AT, and projects TIN2009-13839-C03-01 and TIN2011-27652-C03-01.Alberola Oltra, JM.; Julian Inglada, VJ.; García-Fornes, A. (2014). Challenges for adaptation in agent societies. Knowledge and Information Systems. 38(1):1-34. https://doi.org/10.1007/s10115-012-0565-yS134381Aamodt A, Plaza E (1994) Case-based reasoning; foundational issues, methodological variations, and system approaches. AI Commun 7(1):39–59Abdallah S, Lesser V (2007) Multiagent reinforcement learning and self-organization in a network of agents. In: Proceedings of the sixth international joint conference on autonomous agents and multi-agent systems, pp 172–179Abdu H, Lutfiyya H, Bauer MA (1999) A model for adaptive monitoring configurations. In: Proceedings of the VI IFIP/IEEE IM conference on network management, pp 371–384Alberola JM, Julian V, Garcia-Fornes A (2011) A cost-based transition approach for multiagent systems reorganization. In: Proceedings of the 10th international conference on aut. agents and MAS (AAMAS11), pp 1221–1222Alberola JM, Julian V, Garcia-Fornes A (2012) Multi-dimensional transition deliberation for organization adaptation in multiagent systems. In: Proceedings of the 11th international conference on aut. agents and MAS (AAMAS12) (in press)Argente E, Julian V, Botti V (2006) Multi-agent system development based on organizations. Electron Notes Theor Comput Sci 160(3):55–71Argente E, Botti V, Carrascosa C, Giret A, Julian V, Rebollo M (2011) An abstract architecture for virtual organizations: the Thomas approach. Knowl Inf Syst 29(2):379–403Ashford SJ, Taylor MS (1990) Adaptation to work transitions. An integrative approach. Res Pers Hum Resour Manag 8:1–39Ashford SJ, Blatt R, Walle DV (2003) Reflections on the looking glass: a review of research on feedback-seeking behavior in organizations. J Manag 29(6):773–799Astley WG, Van de Ven AH (1983) Central perspectives and debates in organization theory. Adm Sci Q 28(2):245–273Bond AH, Gasser L (1988) A survey of distributed artificial intelligence readings in distributed artificial intelligence. Morgan Kaufmann, Los AltosBou E, López-Sánchez M, Rodríguez-Aguilar JA (2006) Adaptation of autonomic electronic institutions through norms and institutional agents In: Engineering societies in the agents world. Number LNAI 445, Springer, Dublin, pp 300–319Bou E, López-Sánchez M, Rodríguez-Aguilar JA (2007) Towards self-configuration in autonomic electronic institutions. In: COIN 2006 workshops. Number LNAI 4386, pp 220–235Bou E, López-Sánchez M, Rodríguez-Aguilar JA (2008) Using case-based reasoning in autonomic electronic institutions. In: Proceedings of the 2007 international conference on coordination, organizations, institutions, and norms in agent systems III, pp 125–138Brett JM, Feldman DC, Weingart LR (1990) Feedback-seeking behavior of new hires and job changers. J Manag 16:737–749Bulka B, Gaston ME, desJardins M (2007) Local strategy learning in networked multi-agent team formation. Auton Agents Multi-Agent Syst 15(1):29–45Campos J, López-Sánchez M, Esteva M (2009) Assistance layer, a step forward in multi-agent systems. In: Coordination support international joint conference on autonomous agents and multiagent systems (AAMAS), pp 1301–1302Campos J, Esteva M, López-Sánchez M, Morales J, Salamó M (2011) Organisational adaptation of multi-agent systems in a peer-to-peer scenario. Computing 91(2):169–215Carley KM, and Gasser L (1999) Computational organization theory. Multiagent systems: a modern approach to distributed artificial intelligence. MIT Press, Cambridge, pp 299–330Carvalho G, Almeida H, Gatti M, Vinicius G, Paes R, Perkusich, A, Lucena C (2006) Dynamic law evolution in governance mechanisms for open multi-agent systems. In: Second workshop on software engineering for agent-oriented systemsCernuzzi L, Zambonelli F (2011) Adaptive organizational changes in agent-oriented methodologies. Knowl Eng Rev 26(2):175–190Cheng BH, Lemos R, Giese H, Inverardi P, Magee J (2009) Software engineering for self-adaptive systems: a research roadmap, pp 1–26Corkill DD, Lesser VR (1983) The use of meta-level control for coordination in a distributed problem solving networks. In: Proceedings of the eighth international joint conference on artificial intelligence. IEEE Computer Society Press, pp 748–756Corkill DD, Lander SE (1998) Diversity in agent organizations. Object Mag 8(4):41–47de Paz JF, Bajo J, González A, Rodríguez S, Corchado JM (2012) Combining case-based reasoning systems and support vector regression to evaluate the atmosphere-ocean interaction. Knowl Inf Syst 30(1):155–177DeLoach SA, Matson E (2004) An organizational model for designing adaptive multiagent systems. In: The AAAI-04 workshop on agent organizations: theory and practice (AOTP), pp 66–73DeLoach SA, Oyeman W, Matson E (2008) A capabilities-based model for adaptive organizations. Auton Agents Multi-Agent Syst 16:13–56Dignum V, Dignum F (2001) Modelling agent societies: co-ordination frameworks and institutions progress in artificial intelligence. LNAI 2258, pp 191–204Dignum V (2004) A model for organizational interaction: based on agents, founded in logic. PhD dissertation, Universiteit Utrecht. SIKS dissertation series 2004-1Dignum V, Dignum F, Sonenberg L (2004) Towards dynamic reorganization of agent societies. In: Proceedings of the workshop on coordination in emergent agent societies, pp 22–27Dignum V, Dignum F (2006) Exploring congruence between organizational structure and task performance: a simulation approach coordination, organization, institutions and norms in agent systems I. In: Proceedings of the ANIREM ’05/OOOP ’05, pp 213–230Dignum V, Dignum F (2007) A logic for agent organizations. In: Proceedings of the multi-agent logics, languages, and organisations federated workshops (MALLOW ’007), formal approaches to multi-agent systems (FAMAS ’007) workshopFox MS (1981) Formalizing virtual organizations. IEEE Transact Syst Man Cybern 11(1):70–80Gaston ME, desJardins M (2005) Agent-organized networks for dynamic team formation. In: Proceedings of the fourth international joint conference on autonomous agents and multiagent systems, pp 230–237Gaston ME, desJardins M (2008) The effect of network structure on dynamic team formation in multi-agent systems. Comput Intell 24(2):122–157Norbert G, Philippe M (1997) The reorganization of societies of autonomous agents. In: MAAMAW-97. Springer, London, pp 98–111Goldman CV, Rosenschein JS (1997) Evolving organizations of agents American association for artificial intelligence. In: Multiagent learning workshop at AAAI97Greve HR (1998) Performance, aspirations, and risky organizational change. Adm Sci Quart 43(1):58–86Guessoum Z, Ziane M, Faci N (2004) Monitoring and organizational-level adaptation of multi-agent systems. In: Proceedings of the AAMAS ’04, pp 514–521Hoogendoorn M, Treur J (2006) An adaptive multi-agent organization model based on dynamic role allocation. In: Proceedings of the IAT ’06, pp 474–481Horling B, Benyo B, Lesser V (1999) Using self-diagnosis to adapt organizational structures. In: Proceedings of the 5th international conference on autonomous agents, pp 529–536Horling B, Lesser V (2005) A survey of multi-agent organizational paradigms. Knowl Eng Rev 19(4): 281–316Hrebiniak LG, Joyce WF (1985) Organizational adaptation: strategic choice and environmental determinism. Adm Sci Quart 30(3):336–349Hübner JF, Sichman JS, Boissier O (2002) MOISE+: towards a structural, functional, and deontic model for MAS organization. In: Proceedings of the first international joint conference on autonomous agents and multiagent systems, pp 501–502Hübner JF, Sichman JS, Boissier O (2004) Using the MOISE+ for a cooperative framework of MAS reorganisation. In: Proceedings of the 17th Brazilian symposium on artificial intelligence (SBIA ’04), vol 3171, pp 506–515Hübner JF, Boissier O, Sichman JS (2005) Specifying E-alliance contract dynamics through the MOISE + reorganisation process Anais do V Encontro Nacional de Inteligde Inteligncia Artificial (ENIA 2005)Jennings NR (2001) An agent-based approach for building complex software systems. Commun ACM 44(4):35–41Kamboj S, Decker KS (2006) Organizational self-design in semi-dynamic environments In: 2007 IJCAI workshop on agent organizations: models and simulations (AOMS@IJCAI), pp 335–337Katz D, Kahn RL (1966) The social psychology of organizations. Wiley, New YorkKelly D, Amburgey TL (1991) Organizational inertia and momentum: a dynamic model of strategic change. Acad Manag J 34(3):591–612Kephart J, Chess DM (2003) The vision of autonomic computing. Computer 36(1):41–50Kim DH (1993) The link between individual and organizational learning. Sloan Manag Rev 35(1):37–50Kota R, Gibbins N, Jennings NR (2009a) Decentralised structural adaptation in agent organisations organized adaptation in multi-agent systems, pp 54–71Kota R, Gibbins N, Jennings NR (2009b) Self-organising agent organisations. In: Proceedings of the 8th international conference on autonomous agents and multiagent systems (AAMAS 2009)Kota R, Gibbins N, Jennings NR (2012) Decentralised approaches for self-adaptation in agent organisations. ACM Trans Auton Adapt Syst 7(1):1–28Kotter J, Schlesinger L (1979) Choosing strategies for change. Harv Bus Rev 106–1145Lesser VR (1998) Reflections on the nature of multi-agent coordination and its implications for an agent architecture. Auton Agents Multi-Agent Syst 89–111Levitt B, March JG (1988) Organizational learning. Annu Rev Sociol 14:319–340Luck M, McBurney P, Shehory O, Willmott S (2005) Agent technology: computing as interaction (a roadmap for agent based computing)Mathieu P, Routier JC, Secq Y (2002a) Dynamic organization of multi-agent systems. In: Proceedings of the first international joint conference on autonomous agents and multiagent systems: part 1, pp 451–452Mathieu P, Routier JC, Secq Y (2002b) Principles for dynamic multi-agent organizations. In: Proceedings of the 5th Pacific rim international workshop on multi agents: intelligent agents and multi-agent systems, pp 109–122Matson E, DeLoach S (2003) Using dynamic capability evaluation to organize a team of cooperative, autonomous robots. In: Proceedings of the 2003 international conference on artificial intelligence (IC-AI ’03), Las Vegas, pp 23–26Matson E, DeLoach S (2004) Enabling intra-robotic capabilities adaptation using an organization-based multiagent system. ICRA, pp 2135–2140Matson E, DeLoach S (2005) Formal transition in agent organizations. In: IEEE international conference on knowledge intensive multiagent systems (KIMAS ’05)Matson E, Bhatnagar R (2006) Properties of capability based agent organization transition. In: Proceedings of the IEEE/WIC/ACM international conference on intelligent agent technology IAT ’06, pp 59–65Morales J, López-Sánchez M, Esteva, M (2011) Using experience to generate new regulations. In: Proceedings of the twenty-second international joint conference on artificial Intelligence (IJCAI-11), pp 307–312Muhlestein D, Lim S (2011) Online learning with social computing based interest sharing. Knowl Inf Syst 26(1):31–58Nair R, Tambe M, Marsella S (2003) Role allocation and reallocation in multiagent teams: towards a practical analysis. In: Proceedings of the second AAMAS ’03, pp 552–559Orlikowski WJ (1996) Improvising organizational transformation over time: a situated change perspective. Inf Syst Res 7(1):63–92Panait L, Luke S (2005) Cooperative multi-agent learning: the state of the art. Auton Agents Multi-Agent Syst 11:387–434Ringold PL, Alegria J, Czaplewski RL, Mulder BS, Tolle T, Burnett K (1996) Adaptive monitoring design for ecosystem management. Ecol Appl 6(3):745–747Routier J, Mathieu P, Secq Y (2001) Dynamic skill learning: a support to agent evolution. In: Proceedings of the artificial intelligence and the simulation of behaviour symposium on adaptive agents and multi-agent systems (AISB ’01), pp 25–32Scott RW (2002) Organizations: rational, natural, and open systems, 5th edn. Prentice Hall International, New YorkSeelam A (2009) Reorganization of massive multiagent systems: MOTL/O http://books.google.es/books?id=R-s8cgAACAAJ . Southern Illinois University CarbondaleSo Y, Durfee EH (1993) An organizational self-design model for organizational change. In: AAAI93 workshop on AI and theories of groups and oranizations, pp 8–15So Y, Durfee EH (1998) Designing organizations for computational agents. Simulating organizations. MIT Press, Cambridge, pp 47–64Schwaninger M (2000) A theory for optimal organization. Technical report. Institute of Management at the University of St. Gallen, SwitzerlandTantipathananandh C, Berger-Wolf TY (2011) Finding communities in dynamic social networks. In: IEEE 11th international conference on data mining 2011, pp 1236–1241Wang Z, Liang X (2006) A graph based simulation of reorganization in multi-agent systems. In: IEEE WICACM international conference on intelligent agent technology, pp 129–132Wang D, Tse Q, Zhou Y (2011) A decentralized search engine for dynamic web communities. Knowl Inf Syst 26(1):105–125Weick KE (1979) The social psychology of organizing, 2nd edn. Addison-Wesley, ReadingWeyns D, Haesevoets R, Helleboogh A, Holvoet T, Joosen W (2010a) The MACODO middleware for context-driven dynamic agent organizations. ACM Transact Auton Adapt Syst 3:1–3:28Weyns D, Malek S, Andersson J (2010b) FORMS: a formal reference model for self-adaptation. In: Proceedings of the 7th international conference on autonomic computing, pp 205–214Weyns D, Georgeff M (2010) Self-adaptation using multiagent systems. IEEE Softw 27(1):86–91Zhong C (2006) An investigation of reorganization algorithms. Master-thesi

Agoric computation: trust and cyber-physical systems

In the past two decades advances in miniaturisation and economies of scale have led to the emergence of billions of connected components that have provided both a spur and a blueprint for the development of smart products acting in specialised environments which are uniquely identifiable, localisable, and capable of autonomy. Adopting the computational perspective of multi-agent systems (MAS) as a technological abstraction married with the engineering perspective of cyber-physical systems (CPS) has provided fertile ground for designing, developing and deploying software applications in smart automated context such as manufacturing, power grids, avionics, healthcare and logistics, capable of being decentralised, intelligent, reconfigurable, modular, flexible, robust, adaptive and responsive. Current agent technologies are, however, ill suited for information-based environments, making it difficult to formalise and implement multiagent systems based on inherently dynamical functional concepts such as trust and reliability, which present special challenges when scaling from small to large systems of agents. To overcome such challenges, it is useful to adopt a unified approach which we term agoric computation, integrating logical, mathematical and programming concepts towards the development of agent-based solutions based on recursive, compositional principles, where smaller systems feed via directed information flows into larger hierarchical systems that define their global environment. Considering information as an integral part of the environment naturally defines a web of operations where components of a systems are wired in some way and each set of inputs and outputs are allowed to carry some value. These operations are stateless abstractions and procedures that act on some stateful cells that cumulate partial information, and it is possible to compose such abstractions into higher-level ones, using a publish-and-subscribe interaction model that keeps track of update messages between abstractions and values in the data. In this thesis we review the logical and mathematical basis of such abstractions and take steps towards the software implementation of agoric modelling as a framework for simulation and verification of the reliability of increasingly complex systems, and report on experimental results related to a few select applications, such as stigmergic interaction in mobile robotics, integrating raw data into agent perceptions, trust and trustworthiness in orchestrated open systems, computing the epistemic cost of trust when reasoning in networks of agents seeded with contradictory information, and trust models for distributed ledgers in the Internet of Things (IoT); and provide a roadmap for future developments of our research

Introduction to the Special Issue: The AgentLink III Technical Forums

This article introduces the special issue of ACM Transactions on Autonomous and Adaptive Systems devoted to research papers arising from the three Technical Forum Group meetings held in 2004 and 2005 that were organized and sponsored by the European FP6 Coordination Action AgentLink III