De Raedt, Luc

Demeester, Thomas

Dumancic, S.

Kimmig, A.

Manhaeve, R.

Ghent University Academic Bibliography

Neuro-Symbolic = Neural + Logical + ProbabilisticLuc De Raedt1 , Robin Manhaeve1 , Sebastijan Dumancˇic´1 ,Thomas Demeester2 and Angelika Kimmig31KU Leuven2Ghent University - imec3Cardiff University{luc.deraedt, robin.manhaeve, sebastijan.dumancic}@cs.kuleuven.be,thomas.demeester@ugent.be, KimmigA@cardiff.ac.ukAbstractThe overall goal of neuro-symbolic computationis to integrate high-level reasoning with low-levelperception. We argue 1) that neuro-symbolic com-putation should integrate neural networks with thetwo most prominent methods for reasoning, that is,logic and probability, and 2) that neuro-symbolicintegrated methods should have the pure neural,logical and probabilistic methods as special cases.We examine the state-of-the-art with regard to theseclaims and briefly position our own contributionDeepProbLog in this perspective.1 IntroductionThere is a growing interest in neuro-symbolic computation,that is, in combining high-level reasoning with low-level per-ception that offers the best of both worlds. The growing con-sensus that both forms of reasoning are essential to achievetrue (artificial) intelligence [Kahneman, 2011] has put thequest for neural symbolic computation [Garcez et al., 2012;Garcez et al., 2015; Hammer and Hitzler, 2007] high on theresearch agenda. While deep learning excels at low-level per-ception, there is also a growing awareness of its limitations,certainly in terms of reasoning. Despite various attemptsto demonstrate reasoning-like behaviour with deep learning[Santoro et al., 2017], artificial neural networks’ current rea-soning abilities are nowhere close to what is possible withthe typical high-level reasoning approaches. The two mostprominent frameworks for reasoning are logic and probabil-ity. While in the past, they were studied by separate commu-nities in artificial intelligence, a growing body of researchersis working towards their integration, and even aims at com-bining probability with logic and statistical learning; cf. theareas of statistical relational artificial intelligence [De Raedtet al., 2016; Getoor and Taskar, 2007] and probabilistic logicprogramming [De Raedt and Kimmig, 2015]. The reason-ing abilities of statistical relational artificial intelligence ap-proaches are complementary to the strong pattern-recognitionabilities of deep learning.2 Position statementBased on our experience in upgrading machine learning sys-tems towards the use of (probabilistic) logical and relationalrepresentations [De Raedt, 2008; Muggleton et al., 2012], weargue that a first desirable property of frameworks that inte-grate two other frameworks A and B, is to have the originalframeworks A and B as a special case of the integrated one.If A or B cannot be fully reconstructed, one clearly loses cer-tain abilities, which is not only undesirable but which alsoimplies that there is no true unification. Applying this prop-erty to neuro-symbolic computation implies that the existingframeworks should have both the neural and the symbolicrepresentations as special case. When this is the case, oneretains both the learning abilities of the neural component aswell as the reasoning and learning abilities and the semanticsof the symbolic representations. Unfortunately, this propertyis not satisfied by the vast majority of neuro-symbolic ap-proaches in that they either push the symbolic representationinside the neural network (from which the logic cannot berecovered), or vice versa, apply neural learning principles tosymbolic representations (and risk loosing both the pure neu-ral component and the logical semantics), as we shall show inthe next section.We also advocate a second desirable property for neuro-symbolic computation: models that learn from observed sam-ples should be able to deal with uncertainty. Therefore, oneshould not only integrate logic with neural networks inneuro-symbolic computation, but also probability.This effectively leads to an integration of probabilistic log-ics (hence statistical relational AI) with neural networks andopens up new abilities. Furthermore, although at first sight,this may appear as a complication, it actually can greatlysimplify the integration of neural networks with logic. Thereason for this is that the probabilistic framework providesa clear optimisation criterion, namely the probability of thetraining examples. Real-valued probabilistic quantities arealso well-suited for gradient-based training procedures, as op-posed to discrete logic quantities.3 State-of-the-art in Neuro-SymbolicComputationWe now examine the state-of-the-art in neuro-symbolic com-putation with respect to these two properties, i.e., whetherNeural network /embeddings as special caseLogic asspecial case Probabilistic Underlying logicSemantic based regularization [Diligenti et al., 2017] X FOLLogic Tensor Networks [Donadello et al., 2017] FOLLifted Rule Injection [Demeester et al., 2016] X Implication rulesAdverserial Set Regularisation [Minervini et al., 2017] X Clausal logicSemantic Loss Function [Xu et al., 2018] X X Propositional logic∂ILP [Evans and Grefenstette, 2018] DatalogNeural Theorem Prover [Rockta¨schel and Riedel, 2017] Clausal logicTensorLog [Cohen et al., 2018] X Datalog SubsetDeepProbLog [Manhaeve et al., 2018] X X X Clausal logicTable 1: Comparing Neuro-Symbolic frameworksthey retain the neural network and logic as special cases, andwhether they integrate probabilities. Here, we only considerworks that combine neural networks with logic, although thefield of neuro-symbolic integration is wider than this. Theresults are summarized in Table 1.This subset of the domain can be divided into two cate-gories: logic as constraints and differentiable logic frame-works.Logic as constraints. The dominant idea is to use logicas a constraint on a deep model. That is, the deep model isextended with a regularization term, derived from the desiredlogical properties, which encourages it to mimic logical rea-soning: not obeying such logical properties induces a penalty.For instance, [Diligenti et al., 2017] and [Donadello et al.,2017] use first-order logic to specify constraints on the out-put of the neural network, while [Demeester et al., 2016] and[Minervini et al., 2017] use logical IF-THEN rules, derivedfrom expert knowledge, to enforce the embeddings to be moreconsistent with the logical constraints. [Xu et al., 2018] intro-duce a generalization of this principle such that more complexlogical constraints can be imposed on any deep model. Com-mon to all these approaches is the use of logic as a regular-izer, which encourages deep models to satisfy the constraints.This leads to the logic being encoded into the parameters (ei-ther into the weights of the neural network, or directly intothe embeddings). The constraints are soft, and there is noguarantee that they all will be satisfied. In fact, the trade-offbetween encouraging the constraints vs. following the datain case both are contradictory, is merely a hyperparameter ofthe model. From a logical perspective, constraints only sup-port one form of reasoning, an alternative form of reasoningis based on providing definitions of predicates (in the formof rules or definite clauses), and to use such rules to answerqueries. This is the basis for the popular programming anddatabase languages Prolog and Datalog. The aforementionedmethods thus generally do have neural networks as a specialcase (i.e., when there are no additional constraints), however,they do not have logic as a special case when leaving out theneural part. The importance of being able to fully recover thelogic is already hinted at in some of these papers. For exam-ple, [Xu et al., 2018] show that neural networks trained withthe additional regularization term cannot consistently makepredictions coherent with the logic they were trained on.Differentiable logic frameworks. A different class ofneuro-symbolic systems works by making the logic programdifferentiable, which is achieved by reformulating the basicreasoning primitives using the mathematics of differentiablefunctions. [Rockta¨schel and Riedel, 2017] apply this to thebackward reasoning procedure of Prolog, while [Evans andGrefenstette, 2018] do this in the style of forward reasoning.Although both systems are based on standard reasoning algo-rithms, the original logic cannot be recovered as special case,since the original semantics and inference have been funda-mentally changed. Similarly, [Cohen et al., 2018] introducea framework to compile a tractable subset of logic programsinto differentiable functions and to execute it with neural net-works. All three systems mentioned above construct a neuralnetwork to perform the execution, but they do not have neu-ral networks as a special case. These approaches can onlydeal with rather restricted classes of logic programs. Further-more, because they push the logic programming componentsinto the neural networks, both the proper semantics of logicprograms and the programming language nature of the frame-work are sacrificed. This is clear when considering that theapproach of [Rockta¨schel and Riedel, 2017] is to map a prooftree on a neural network, which can then be differentiated asthe logic has been removed. These systems also suffer fromnot keeping the original logic as a special case. For example,[Rockta¨schel and Riedel, 2017] and [Evans and Grefenstette,2018] both mention that the neural execution of the differen-tiable logic creates a large computational overhead.4 Our approach: DeepProbLogThe proposed approach is pursued in DeepProbLog [Man-haeve et al., 2018], a neuro-symbolic framework developedin the past year. What sets DeepProbLog apart from theother approaches mentioned above is that instead of chang-ing the logic, it is extended. To create DeepProbLog,we extended the probabilistic logic programming languageProbLog [Fierens et al., 2015] with neural predicates. Theunderlying concept of a probabilistic logic programming lan-guage is simple: (ground) atomic expressions of the formq(t1, ..., tn) (aka tuples in a relational database) are consid-ered as (independent) random variables that have a probabil-ity p of being true. This view, which was pioneered in the dis-tribution semantics by [Sato, 1995], effectively unifies the ba-sic primitives in logic with those in probability theory: propo-sitions become random variables. On top of these “proba-bilistic facts” one can then define rules (in the form of logicprograms) that allow to derive conclusions from these factsand induce a probability distribution over possible worlds.The neural predicate extends this concept so that atomic ex-pressions can also be annotated with the output of a neuralnetwork, given that the output can be considered a probability.This simple idea is appealing as it allows us to retain all theessential components of the probabilistic logic programminglanguage: the semantics, the inference mechanism, as well asthe implementation. At the same time, it clearly decouplesthe logical component from the neural component. As neu-ral networks become predicates, they can be called from thelogic programs and so the logical layer is at the higher-leveland the neural one at the lower-level with the neural predi-cates providing the interface, while both sides can treat eachother as a black box. For example, consider the task wherewe have to learn to classify sums from pairs of handwrittendigits, e.g + = 8. In line with our position statement,the neural network needs to handle only the image recogni-tion, while the addition is handled purely by logic. This taskcan thus easily be solved in DeepProbLog by specifying aneural predicate that defines the classification of single dig-its, and a single line of logic that defines the addition. Asshown in [Manhaeve et al., 2018], this single line of logic isenough to clearly outperform a fully neural baseline, as it con-verges quicker, to a higher accuracy. Furthermore, becausethe neural network in the DeepProbLog program classifiesdigits, it can be reused for other tasks after training, whereasthe baseline can only be used on the same task. It is easy tosee how both the logic and the neural networks are preservedas special cases: if the DeepProbLog program has no neu-ral predicates, it simply becomes a normal ProbLog program.If that program has no probabilistic facts, it becomes a purelogic program. Furthermore, if the program only containsa neural predicate, it becomes equivalent to directly trainingthe neural network. In describing the learning algorithm forDeepProbLog it is important to note that DeepProbLog, incontrast to other approaches for neuro-symbolic learning, notonly encompasses a neural and a logical framework, but alsoa probabilistic one. Making the logic probabilistic also makesit differentiable, allowing gradients to be derived so that theneural networks can be trained with standard gradient descentmethods. For the given example, note that the neural predi-cate is trained from an indirect signal, i.e., from observationsof sums (at the output of the logical layer), rather than throughlabels of individual digits.To summarize, DeepProbLog occupies a unique positionin the space of neuro-symbolic methods in that i) it is a pro-gramming language that supports neural networks and ma-chine learning, and it has a well-defined semantics (as an ex-tension of Prolog, it is Turing equivalent); (ii) it integrateslogical reasoning with neural networks; so both symbolic andsubsymbolic representations and inference, learning and rea-soning, perception and inference; (iii) it integrates probabilis-tic modeling, programming and reasoning with neural net-works (as DeepProbLog extends the probabilistic program-ming language ProbLog, which can be regarded as a very ex-pressive directed graphical modeling language [De Raedt etal., 2016]).AcknowledgementsThis work has been partially supported by the European Re-search Council Advanced Grant project SYNTH9 (ERCAdG-694980). RM is a SB PhD fellow at FWO (1S61718N).References[Cohen et al., 2018] William W Cohen, Fan Yang, andKathryn Rivard Mazaitis. Tensorlog: Deep learning meetsprobabilistic databases. Journal of Artificial IntelligenceResearch, 1:1–15, 2018.[De Raedt and Kimmig, 2015] Luc De Raedt and AngelikaKimmig. Probabilistic (logic) programming concepts. Ma-chine Learning, 100(1):1–43, 2015.[De Raedt et al., 2016] Luc De Raedt, Kristian Kersting, Sri-raam Natarajan, and David Poole. Statistical relational ar-tificial intelligence: Logic, probability and computation.Synthesis Lectures on Artificial Intelligence and MachineLearning, 2016.[De Raedt, 2008] Luc De Raedt. Logical and RelationalLearning. Springer, 2008.[Demeester et al., 2016] Thomas Demeester,T. Rockta¨schel, and S. Riedel. Lifted rule injectionfor relation embeddings. In EMNLP, 2016.[Diligenti et al., 2017] Michelangelo Diligenti, Marco Gori,and Claudio Sacca. Semantic-based regularization forlearning and inference. Artificial Intelligence, 244:143–165, 2017.[Donadello et al., 2017] Ivan Donadello, Luciano Serafini,and Artur S. d’Avila Garcez. Logic tensor networks forsemantic image interpretation. In IJCAI, 2017.[Evans and Grefenstette, 2018] Richard Evans and EdwardGrefenstette. Learning explanatory rules from noisy data.JAIR, 2018.[Fierens et al., 2015] Daan Fierens, Guy Van den Broeck,Joris Renkens, Dimitar Shterionov, Bernd Gutmann, IngoThon, Gerda Janssens, and Luc De Raedt. Inference andlearning in probabilistic logic programs using weightedBoolean formulas. Theory and Practice of Logic Program-ming, 15(3):358–401, 2015.[Garcez et al., 2012] Artur S d’Avila Garcez, Krysia BBroda, and Dov M Gabbay. Neural-symbolic learning sys-tems: foundations and applications. Springer Science &Business Media, 2012.[Garcez et al., 2015] Artur d’Avila Garcez, Tarek R Besold,Luc De Raedt, Peter Fo¨ldiak, Pascal Hitzler, ThomasIcard, Kai-Uwe Ku¨hnberger, Luis C Lamb, Risto Miikku-lainen, and Daniel L Silver. Neural-symbolic learning andreasoning: contributions and challenges. In 2015 AAAISpring Symposium Series, 2015.[Getoor and Taskar, 2007] Lise Getoor and Ben Taskar. In-troduction to statistical relational learning. MIT press,2007.[Hammer and Hitzler, 2007] Barbara Hammer and PascalHitzler. Perspectives of neural-symbolic integration, vol-ume 8. Springer Heidelberg:, 2007.[Kahneman, 2011] Daniel Kahneman. Thinking, fast andslow. Farrar, Straus and Giroux New York, 2011.[Manhaeve et al., 2018] Robin Manhaeve, Sebastijan Du-mancic, Angelika Kimmig, Thomas Demeester, and LucDe Raedt. Deepproblog: Neural probabilistic logic pro-gramming. In NeurIPS, 2018.[Minervini et al., 2017] Pasquale Minervini, Thomas De-meester, Tim Rockta¨schel, and Sebastian Riedel. Adver-sarial sets for regularising neural link predictors. In UAI,2017.[Muggleton et al., 2012] Stephen Muggleton, Luc De Raedt,David Poole, Ivan Bratko, Peter Flach, Katsumi Inoue,and Ashwin Srinivasan. Ilp turns 20. Machine learning,86(1):3–23, 2012.[Rockta¨schel and Riedel, 2017] Tim Rockta¨schel and Sebas-tian Riedel. End-to-end differentiable proving. In NIPS,2017.[Santoro et al., 2017] Adam Santoro, David Raposo,David G Barrett, Mateusz Malinowski, Razvan Pascanu,Peter Battaglia, and Tim Lillicrap. A simple neuralnetwork module for relational reasoning. In NIPS, 2017.[Sato, 1995] T. Sato. A statistical learning method for logicprograms with distribution semantics. In ICLP, 1995.[Xu et al., 2018] Jingyi Xu, Zilu Zhang, Tal Friedman, YitaoLiang, and Guy Van den Broeck. A semantic loss func-tion for deep learning with symbolic knowledge. In ICML,2018.

Neuro-Symbolic = Neural + Logical + Probabilistic

https://biblio.ugent.be/publication/8618135/file/8618137.pdf

Neuro-Symbolic = Neural + Logical + Probabilistic

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography