Context Meta-Reinforcement Learning via Neuromodulation

Meta-reinforcement learning (meta-RL) algorithms enable agents to adapt quickly to tasks from few samples in dynamic environments. Such a feat is achieved through dynamic representations in an agent's policy network (obtained via reasoning about task context, model parameter updates, or both). However, obtaining rich dynamic representations for fast adaptation beyond simple benchmark problems is challenging due to the burden placed on the policy network to accommodate different policies. This paper addresses the challenge by introducing neuromodulation as a modular component to augment a standard policy network that regulates neuronal activities in order to produce efficient dynamic representations for task adaptation. The proposed extension to the policy network is evaluated across multiple discrete and continuous control environments of increasing complexity. To prove the generality and benefits of the extension in meta-RL, the neuromodulated network was applied to two state-of-the-art meta-RL algorithms (CAVIA and PEARL). The result demonstrates that meta-RL augmented with neuromodulation produces significantly better result and richer dynamic representations in comparison to the baselines

Using Event Calculus to Formalise Policy Specification and Analysis

As the interest in using policy-based approaches for systems management grows, it is becoming increasingly important to develop methods for performing analysis and refinement of policy specifications. Although this is an area that researchers have devoted some attention to, none of the proposed solutions address the issues of analysing specifications that combine authorisation and management policies; analysing policy specifications that contain constraints on the applicability of the policies; and performing a priori analysis of the specification that will both detect the presence of inconsistencies and explain the situations in which the conflict will occur. We present a method for transforming both policy and system behaviour specifications into a formal notation that is based on event calculus. Additionally it describes how this formalism can be used in conjunction with abductive reasoning techniques to perform a priori analysis of policy specifications for the various conflict types identified in the literature. Finally, it presents some initial thoughts on how this notation and analysis technique could be used to perform policy refinement

A Formal Framework for Concrete Reputation Systems

In a reputation-based trust-management system, agents maintain information about the past behaviour of other agents. This information is used to guide future trust-based decisions about interaction. However, while trust management is a component in security decision-making, many existing reputation-based trust-management systems provide no formal security-guarantees. In this extended abstract, we describe a mathematical framework for a class of simple reputation-based systems. In these systems, decisions about interaction are taken based on policies that are exact requirements on agents’ past histories. We present a basic declarative language, based on pure-past linear temporal logic, intended for writing simple policies. While the basic language is reasonably expressive (encoding e.g. Chinese Wall policies) we show how one can extend it with quantification and parameterized events. This allows us to encode other policies known from the literature, e.g., ‘one-out-of-k’. The problem of checking a history with respect to a policy is efficient for the basic language, and tractable for the quantified language when policies do not have too many variables

Semantic-based policy engineering for autonomic systems

This paper presents some important directions in the use of ontology-based semantics in achieving the vision of Autonomic Communications. We examine the requirements of Autonomic Communication with a focus on the demanding needs of ubiquitous computing environments, with an emphasis on the requirements shared with Autonomic Computing. We observe that ontologies provide a strong mechanism for addressing the heterogeneity in user task requirements, managed resources, services and context. We then present two complimentary approaches that exploit ontology-based knowledge in support of autonomic communications: service-oriented models for policy engineering and dynamic semantic queries using content-based networks. The paper concludes with a discussion of the major research challenges such approaches raise