Towards generalizable neuro-symbolic reasoners

Doctor of PhilosophyDepartment of Computer ScienceMajor Professor Not ListedSymbolic knowledge representation and reasoning and deep learning are fundamentally different approaches to artificial intelligence with complementary capabilities. The former are transparent and data-efficient, but they are sensitive to noise and cannot be applied to non-symbolic domains where the data is ambiguous. The latter can learn complex tasks from examples, are robust to noise, but are black boxes; require large amounts of --not necessarily easily obtained-- data, and are slow to learn and prone to adversarial examples. Either paradigm excels at certain types of problems where the other paradigm performs poorly. In order to develop stronger AI systems, integrated neuro-symbolic systems that combine artificial neural networks and symbolic reasoning are being sought. In this context, one of the fundamental open problems is how to perform logic-based deductive reasoning over knowledge bases by means of trainable artificial neural networks. Over the course of this dissertation, we provide a brief summary of our recent efforts to bridge the neural and symbolic divide in the context of deep deductive reasoners. More specifically, We designed a novel way of conducting neuro-symbolic through pointing to the input elements. More importantly we showed that the proposed approach is generalizable across new domain and vocabulary demonstrating symbol-invariant zero-shot reasoning capability. Furthermore, We have demonstrated that a deep learning architecture based on memory networks and pre-embedding normalization is capable of learning how to perform deductive reason over previously unseen RDF KGs with high accuracy. We are applying these models on Resource Description Framework (RDF), first-order logic, and the description logic EL+ respectively. Throughout this dissertation we will discuss strengths and limitations of these models particularly in term of accuracy, scalability, transferability, and generalizabiliy. Based on our experimental results, pointer networks perform remarkably well across multiple reasoning tasks while outperforming the previously reported state of the art by a significant margin. We observe that the Pointer Networks preserve their performance even when challenged with knowledge graphs of the domain/vocabulary it has never encountered before. To our knowledge, this work is the first attempt to reveal the impressive power of pointer networks for conducting deductive reasoning. Similarly, we show that memory networks can be trained to perform deductive RDFS reasoning with high precision and recall. The trained memory network's capabilities in fact transfer to previously unseen knowledge bases. Finally will talk about possible modifications to enhance desirable capabilities. Altogether, these research topics, resulted in a methodology for symbol-invariant neuro-symbolic reasoning

Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning

Learning the underlying patterns in data goes beyondinstance-based generalization to external knowledge repre-sented in structured graphs or networks. Deep learning thatprimarily constitutes neural computing stream in AI hasshown significant advances in probabilistically learning la-tent patterns using a multi-layered network of computationalnodes (i.e., neurons/hidden units). Structured knowledge thatunderlies symbolic computing approaches and often supportsreasoning, has also seen significant growth in recent years,in the form of broad-based (e.g., DBPedia, Yago) and do-main, industry or application specific knowledge graphs. Acommon substrate with careful integration of the two willraise opportunities to develop neuro-symbolic learning ap-proaches for AI, where conceptual and probabilistic repre-sentations are combined. As the incorporation of externalknowledge will aid in supervising the learning of features forthe model, deep infusion of representational knowledge fromknowledge graphs within hidden layers will further enhancethe learning process. Although much work remains, we be-lieve that knowledge graphs will play an increasing role in de-veloping hybrid neuro-symbolic intelligent systems (bottom-up deep learning with top-down symbolic computing) as wellas in building explainable AI systems for which knowledgegraphs will provide scaffolding for punctuating neural com-puting. In this position paper, we describe our motivation forsuch a neuro-symbolic approach and framework that com-bines knowledge graph and neural networks

Logic Programming and Machine Ethics

Transparency is a key requirement for ethical machines. Verified ethical behavior is not enough to establish justified trust in autonomous intelligent agents: it needs to be supported by the ability to explain decisions. Logic Programming (LP) has a great potential for developing such perspective ethical systems, as in fact logic rules are easily comprehensible by humans. Furthermore, LP is able to model causality, which is crucial for ethical decision making.Comment: In Proceedings ICLP 2020, arXiv:2009.09158. Invited paper for the ICLP2020 Panel on "Machine Ethics". arXiv admin note: text overlap with arXiv:1909.0825

Is depression related to cannabis? : A Knowledge-infused Model for Entity and Relation Extraction with Limited Supervision

With strong marketing advocacy of the benefits of cannabis use for improved mental health, cannabis legalization is a priority among legislators. However, preliminary scientific research does not conclusively associate cannabis with improved mental health. In this study, we explore the relationship between depression and consumption of cannabis in a targeted social media corpus involving personal use of cannabis with the intent to derive its potential mental health benefit. We use tweets that contain an association among three categories annotated by domain experts - Reason, Effect, and Addiction. The state-of-the-art Natural Langauge Processing techniques fall short in extracting these relationships between cannabis phrases and the depression indicators. We seek to address the limitation by using domain knowledge; specifically, the Drug Abuse Ontology for addiction augmented with Diagnostic and Statistical Manual of Mental Disorders lexicons for mental health. Because of the lack of annotations due to the limited availability of the domain experts’ time, we use supervised contrastive learning in conjunction with GPT-3 trained on a vast corpus to achieve improved performance even with limited supervision. Experimental results show that our method can significantly extract cannabis-depression relationships better than the state-of-the-art relation extractor. High-quality annotations can be provided using a nearest neighbor approach using the learned representations that can be used by the scientific community to understand the association between cannabis and depression better

Framing TRUST in Artificial Intelligence (AI) Ethics Communication: Analysis of AI Ethics Guiding Principles through the Lens of Framing Theory

With the fast proliferation of Artificial Intelligence (AI) technologies in our society, several corporations, governments, research institutions, and NGOs have produced and published AI ethics guiding documents. These include principles, guidelines, frameworks, assessment lists, training modules, blogs, and principle-to-practice strategies. The priorities, focus, and articulation of these innumerable documents vary to different extents. Though they all aim and claim to ensure AI usage for the common good, the actual AI system outcomes in various social applications have invigorated ethical dilemmas and scholarly debates. This study presents the analysis of AI ethics principles and guidelines text published by three pioneers from three different sectors - Microsoft Corporation, National Institute of Standards and Technology (NIST), AI HLEG set up by the European Commission through the lens of media and communication’s Framing Theory. The TRUST Framings extracted from recent academic AI literature are used as standard construct to study the ethics framings in the selected text. The institutional framing of AI principles and guidelines shapes the AI ethics of an institution in a soft (as there is no legal binding) but strong (incorporating their respective position/societal role’s priorities) way. The AI principles’ framing approach directly relates to the AI actor’s ethics that enjoins risk mitigation and problem resolution associated with AI development and deployment cycle. Thus, it has become important to examine institutional AI ethics communication. This paper brings forth a Comm-Tech perspective around the ethics of evolving technologies known under the umbrella term - Artificial Intelligence and the human moralities governing them

Decision support combining machine learning, knowledge representation and case-based reasoning

Knowledge and knowledge work are essential for the success of companies nowadays. Decisions are based on knowledge and better knowledge leads to more informed decisions. Therefore, the management of knowledge and support of decision making has increasingly become a source of competitive advantage for organizations. The current research uses a design science research approach (DSR) with the aim to improve the decision making of a knowledge intensive process such as the student admission process, which is done manually until now. In the awareness phase of the DSR process, the case study research method is applied to analyze the decision making and the knowledge that is needed to derive the decisions. Based on the analysis of the application scenario, suitable methods to support decision making were identified. The resulting system design is based on a combination of Case-Based Reasoning (CBR) and Machine Learning (ML). The proposed system design and prototype has been validated using triangulation evaluation, to assess the impact of the proposed system on the application scenario. The evaluation revealed that the additional hints from CBR and ML can assist the deans of the study program to improve the knowledge management and increase the quality, transparency and consistency of the decision-making process in the student application process. Furthermore, the proposed approach fosters the exchange of knowledge among the different process participants involved and codifies previously tacit knowledge to some extent and provides relevant externalized knowledge to decision makers at the required moment. The designed prototype showcases how ML and CBR methodologies can be combined to support decision making in knowledge intensive processes and finally concludes with potential recommendations for future research.http://ceur-ws.orgam2022Informatic

"Is depression related to cannabis?": A knowledge-infused model for Entity and Relation Extraction with Limited Supervision

With strong marketing advocacy of the benefits of cannabis use for improved mental health, cannabis legalization is a priority among legislators. However, preliminary scientific research does not conclusively associate cannabis with improved mental health. In this study, we explore the relationship between depression and consumption of cannabis in a targeted social media corpus involving personal use of cannabis with the intent to derive its potential mental health benefit. We use tweets that contain an association among three categories annotated by domain experts - Reason, Effect, and Addiction. The state-of-the-art Natural Langauge Processing techniques fall short in extracting these relationships between cannabis phrases and the depression indicators. We seek to address the limitation by using domain knowledge; specifically, the Drug Abuse Ontology for addiction augmented with Diagnostic and Statistical Manual of Mental Disorders lexicons for mental health. Because of the lack of annotations due to the limited availability of the domain experts' time, we use supervised contrastive learning in conjunction with GPT-3 trained on a vast corpus to achieve improved performance even with limited supervision. Experimental results show that our method can significantly extract cannabis-depression relationships better than the state-of-the-art relation extractor. High-quality annotations can be provided using a nearest neighbor approach using the learned representations that can be used by the scientific community to understand the association between cannabis and depression better.Comment: Accepted to AAAI-2021 Symposiu

Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning

Learning the underlying patterns in data goes beyond instance-based generalization to external knowledge represented in structured graphs or networks. Deep learning that primarily constitutes neural computing stream in AI has shown significant advances in probabilistically learning latent patterns using a multi-layered network of computational nodes (i.e., neurons/hidden units). Structured knowledge that underlies symbolic computing approaches and often supports reasoning, has also seen significant growth in recent years, in the form of broad-based (e.g., DBPedia, Yago) and domain, industry or application specific knowledge graphs. A common substrate with careful integration of the two will raise opportunities to develop neuro-symbolic learning approaches for AI, where conceptual and probabilistic representations are combined. As the incorporation of external knowledge will aid in supervising the learning of features for the model, deep infusion of representational knowledge from knowledge graphs within hidden layers will further enhance the learning process. Although much work remains, we believe that knowledge graphs will play an increasing role in developing hybrid neuro-symbolic intelligent systems (bottom-up deep learning with top-down symbolic computing) as well as in building explainable AI systems for which knowledge graphs will provide scaffolding for punctuating neural computing. In this position paper, we describe our motivation for such a neuro-symbolic approach and framework that combines knowledge graph and neural networks