End-to-End Reinforcement Learning for Automatic Taxonomy Induction

We present a novel end-to-end reinforcement learning approach to automatic taxonomy induction from a set of terms. While prior methods treat the problem as a two-phase task (i.e., detecting hypernymy pairs followed by organizing these pairs into a tree-structured hierarchy), we argue that such two-phase methods may suffer from error propagation, and cannot effectively optimize metrics that capture the holistic structure of a taxonomy. In our approach, the representations of term pairs are learned using multiple sources of information and used to determine \textit{which} term to select and \textit{where} to place it on the taxonomy via a policy network. All components are trained in an end-to-end manner with cumulative rewards, measured by a holistic tree metric over the training taxonomies. Experiments on two public datasets of different domains show that our approach outperforms prior state-of-the-art taxonomy induction methods up to 19.6\% on ancestor F1.Comment: 11 Pages. ACL 2018 Camera Read

QDQD-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations

The paper considers a class of multi-agent Markov decision processes (MDPs), in which the network agents respond differently (as manifested by the instantaneous one-stage random costs) to a global controlled state and the control actions of a remote controller. The paper investigates a distributed reinforcement learning setup with no prior information on the global state transition and local agent cost statistics. Specifically, with the agents' objective consisting of minimizing a network-averaged infinite horizon discounted cost, the paper proposes a distributed version of $Q$-learning, $\mathcal{QD}$-learning, in which the network agents collaborate by means of local processing and mutual information exchange over a sparse (possibly stochastic) communication network to achieve the network goal. Under the assumption that each agent is only aware of its local online cost data and the inter-agent communication network is \emph{weakly} connected, the proposed distributed scheme is almost surely (a.s.) shown to yield asymptotically the desired value function and the optimal stationary control policy at each network agent. The analytical techniques developed in the paper to address the mixed time-scale stochastic dynamics of the \emph{consensus + innovations} form, which arise as a result of the proposed interactive distributed scheme, are of independent interest.Comment: Submitted to the IEEE Transactions on Signal Processing, 33 page

Modular Design Patterns for Hybrid Learning and Reasoning Systems: a taxonomy, patterns and use cases

The unification of statistical (data-driven) and symbolic (knowledge-driven) methods is widely recognised as one of the key challenges of modern AI. Recent years have seen large number of publications on such hybrid neuro-symbolic AI systems. That rapidly growing literature is highly diverse and mostly empirical, and is lacking a unifying view of the large variety of these hybrid systems. In this paper we analyse a large body of recent literature and we propose a set of modular design patterns for such hybrid, neuro-symbolic systems. We are able to describe the architecture of a very large number of hybrid systems by composing only a small set of elementary patterns as building blocks. The main contributions of this paper are: 1) a taxonomically organised vocabulary to describe both processes and data structures used in hybrid systems; 2) a set of 15+ design patterns for hybrid AI systems, organised in a set of elementary patterns and a set of compositional patterns; 3) an application of these design patterns in two realistic use-cases for hybrid AI systems. Our patterns reveal similarities between systems that were not recognised until now. Finally, our design patterns extend and refine Kautz' earlier attempt at categorising neuro-symbolic architectures.Comment: 20 pages, 22 figures, accepted for publication in the International Journal of Applied Intelligenc

Depressive statements prime goal-directed alcohol-seeking in individuals who report drinking to cope with negative affect

This is the author accepted manuscript. The final version is available from Springer Verlag via the DOI in this record.Background Most variants of negative reinforcement theory predict that acute depressed mood can promote alcohol-seeking behaviour, but the precise mechanisms underpinning this effect remain contested. One possibility is that mood-induced alcohol-seeking is due to the formation of a stimulus-response (S-R) association, enabling depressed mood to elicit alcohol-seeking automatically. A second possibility is that depressed mood undergoes incentive learning, enabling it to enhance the expected value of alcohol and thus promote goal-directed alcohol-seeking. Objectives These two explanations were distinguished using a human outcome-revaluation procedure. Methods One hundred and twenty eight alcohol drinkers completed questionnaires of alcohol use disorder, drinking to cope with negative affect and depression symptoms. Participants then learned that two responses earned alcohol and food points respectively (baseline) in two-alternative forced-choice trials. At test, participants rated the valence of randomly sampled negative and positive mood statements and, after each statement, chose between the alcohol- or food-seeking response in extinction. Results The percentage of alcohol- vs. food-seeking responses was increased significantly in trials containing negative statements compared to baseline and positive statement trials, in individuals who reported drinking to cope with negative affect (p=.004), but there was no such interaction with indices of alcohol use disorder (p=.87) or depression symptoms (p=.58). Conclusions: Individuals who drink to cope with negative affect are more sensitive to the motivational impact of acute depressed mood statements priming goal-directed alcoholseeking. Negative copers’ vulnerability to alcohol dependence may be better explained by excessive affective incentive learning than by S-R habit formation.The work was supported by the ESRC and Alcohol Research UK