Hierarchical reinforcement learning for communicating agents

This paper proposes hierarchical reinforcement learning (RL) methods for communication in multiagent coordination problems modelled as Markov Decision Processes (MDPs). To bridge the gap between the MDP view and the methods used to specify communication protocols in multiagent systems (using logical conditions and propositional message structure), we utilise interaction frames as powerful policy abstractions that can be combined with case-based reasoning techniques. Also, we exploit the fact that breaking communication processes down to manageable “chunks ” of interaction sequences (as suggested by the interaction frames approach) naturally corresponds to methods proposed in the area of hierarchical RL. The approach is illustrated and validated through experiments in a complex application domain which prove that it is capable of handling large state and action spaces.

On the Mutability of Protocols

Centre for Intelligent Systems and their ApplicationsThe task of developing a framework for which agents can communicate reliably and flexibly in open systems is not trivial. This thesis addresses the dichotomy between reliable communication and facilitation of the autonomy of agents to create more flexible and emergent interactions. By the introduction of adaptations to a distributed protocol language, agents benefit from the ability to communicate interaction protocols to elucidate the social norms (thus creating more reliable communication). Yet, this approach also provides the functionality for the agent to unilaterally introduce new paths for the conversation to explore unforeseen opportunities and options (thus restoring more autonomy than possible with static protocols). The foundation of this work is Lightweight Coordination Calculus (LCC). LCC is a distributed protocol language and framework in which agents coordinate their own interactions by their message passing activities. In order to ensure that adaptations to the protocols are done in a reasonable way, we examine the use of two models of communication to guide any transformations to the protocols. We describe the use of FIPA's ACL and ultimately its unsuitability for this approach as well as the more fecund task of implementing dialogue games, an model of argumentation, as dynamic protocols. The existing attempts to develop a model that can encompass the gulf between reliability and autonomy in communication have had varying degrees of success. It is the purpose and the result of the research described in this thesis to develop an alloy of the various models, by the introduction of dynamic and distributed protocols, to develop a framework stronger than its constituents. Though this is successful, the derivations of the protocols can be dificult to reconstruct. To this end, this thesis also describes a method of protocol synthesis inspired by models of human communication that can express the dialogues created by the previous approaches but also have a fully accountable path of construction. Not only does this thesis explore a unique and novel approach to agent communication, it is tested through a practical implementation