Csiszárik, Adrián

Kaliszyk, Cezary

Michalewski, Henryk

Urban, Josef

Zombori, Zsolt

Repository of the Academy's Library

Update on FLoP, a Reinforcement Learning based TheoremProver ∗Zsolt Zombori1, Adrián Csiszárik1, Henryk Michalewski3, Cezary Kaliszyk2, andJosef Urban41 Alfréd Rényi Institute of Mathematics, Budapest2 University of Innsbruck3 University of Warsaw, Google4 Czech Technical University in Prague1 IntroductionThe FLoP system was built to allow for experimenting with advanced reinforcement learning(RL) methods applied to guide theorem proving. Its particular focus is to enable learning fromand generalizing to long proofs, which is a largely unsolved challenge in theorem proving. Thesystem is very flexible in terms of what it can learn from: even a single training environment(proof) can result in meaningful generalization. On the other hand, FLoP is simplistic in severalways: 1) it learns from manually extracted features, 2) it can overfit in some learning scenariosand 3) its merits have so far been demonstrated only on a very simple dataset. Here we onlyaddress 1), the problem of feature extraction.We present ongoing work that aims to use graph neural networks (gnn) [9] for feature extraction.Gnns have been used to learn features of logic formulae on several supervised tasks, e.g. [3, 7,8, 2]. However, there are very few experiments with such extractors in a reinforcement learningsetting. RL models are typically convolutional and dense networks. Related exceptions are [6]and [5] that use graph extractors. However, while these systems use intertwined iterations ofproof search and supervised learning, FLoP uses a pure reinforcement learning loop.We consider learned formula embedding as a stepping stone for more involved projects thatcombine machine learning and theorem proving. In Appendix A and B we briefly present twosuch project proposals planned as future work.2 Feature extractionMachine learning models require inputs embedded into some Euclidean space Rn. However,when it comes to learning to guide a theorem prover, states and actions are given as logicalformulae and it is highly unclear how to turn them into fixed length vectors. An often usedapproach is to do some manual feature extraction. Currently, FLoP extracts triples of adja-cent nodes in the formula trees as features. These features convey some statistically relevant∗ZZ and AC were supported by the European Union, co-financed by the European Social Fund (EFOP-3.6.3-VEKOP-16-2017-00002) and the Hungarian National Excellence Grant 2018-1.2.1-NKP-00008. HM wassupported by the Polish National Science Center grant UMO-2018/29/B/ST6/02959. CK was supported byERC grant no. 714034 SMART. JU was funded by the AI4REASON ERC Consolidator grant nr. 649043,the Czech project AI&Reasoning CZ.02.1.01/0.0/0.0/15 003/0000466 and the European Regional DevelopmentFund.Update on FLoP Zombori, Csiszárik, Michalewski, Kaliszyk, Urbaninformation, however, a large part of the semantics is lost. Another approach that is gainingpopularity is to represent formulae as graphs and use graph neural networks to produce an em-bedding vector. Their promise is to adapt feature extraction both to the data and the problem,i.e., to produce an embedding that best fits the current learning task.3 Embedding with Graph Neural NetworksA gnn takes a labelled graph as input. Each node has some inital embedding vector. The initialembedding is refined in multiple iterations using a learnable updater model : the new embeddingis calculated from previous embeddings of its neighbourhood. Hence, it exploits the structureof the graph, allowing information to propagate along the edges. We perform a fixed numberof update operations, called hops to obtain the final embedding of the nodes. Finally, someaggregation operation creates a single embedding of the graph from that of its nodes.Projects using gnns show a large variance with respect to how the input is turned into a graph.We present our proposed approach by comparing it to two recent variants: [7] and [8, 6].NeuroSAT [7] embeds propositional formulae in conjunctive normal form (CNF). The result-ing graph has 2 kinds of nodes (clauses, literals) and 2 kinds of edges (from literals to their con-taining clauses, between negated literal pairs). Thanks to the small number of node/edge labels,each kind of interaction is represented by separate neural networks in the update step.FormulaNet and Graph Embeddings for HOList [8, 6] embed formulae of higher orderlogic. The graph is the abstract syntax tree of the formula. The number of different symbols thatcan occur in the input is not bounded, so a single update operation is performed on all nodes.Node type information is preserved in a learnable initial embedding. Function application iscurried in the syntax tree, so each node has at most two children, i.e., we only need two types ofedges. Identical subexpressions are merged. A major complication, that was not present in [7]is the representation of variables. [8, 6] collapse all variables into a single ”VAR” symbol.FLoP In FLoP, we embed first order formulae in CNF. Our implementation is very similarto that of [6], we start from the syntax tree, with two differences: 1) The initial embeddingis a fixed random vector. 2) Variables are not collapsed into a single node. Rather, they arewrapped into a ”VAR” function and are normalised to ensure that they are renaming invariant.This setup ensures that the formula is recoverable from the graph: the initial embedding vectorof the nth variable (according to a preorder traversal) is the same for all inputs.4 Graph Embedding in FLoPFLoP is built on the leanCoP connection tableau calculus, so its current state is given by the setof valid actions and the partial tableau tree, with the following main components: the currentgoal, the branch leading to the current goal, remaining open goals and the currently applicablelemmas. At each step, a policy network computes a probability distribution over the valid2Update on FLoP Zombori, Csiszárik, Michalewski, Kaliszyk, Urbanactions. Each component of the input is currently a hand crafted feature vector, which can beeasily replaced with the embedding network described above.This is an ongoing effort and in our talk we will present first results using graph embeddings inFLoP. As a proof of concept, we have done some supervised experiments that use our embeddingnetwork: we collected theorem proving attempts from FLoP training and trained to predict if a(state, action) pair can lead to success. We achieved 100% training accuracy on a training setof 20000 entries. We are working to see how well it generalizes.References[1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, BobMcGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. CoRR,abs/1707.01495, 2017.[2] Karel Chvalovsky. Top-down neural model for formulae. In International Conference on LearningRepresentations, 2019.[3] Karel Chvalovský, Jan Jakubuv, Martin Suda, and Josef Urban. Enigma-ng: Efficient neural andgradient-boosted inference guidance for e. In CADE, 2019.[4] The MPTP Challenge. http://www.tptp.org/Seminars/MizarVerification/TheMPTPChallenge.html. Accessed: 2019-05-20.[5] Miroslav Oľsák, Cezary Kaliszyk, and Josef Urban. Property invariant embedding for automatedreasoning. CoRR, 2019.[6] Aditya Paliwal, Sarah M. Loos, Markus N. Rabe, Kshitij Bansal, and Christian Szegedy. Graphrepresentations for higher-order logic and theorem proving. CoRR, abs/1905.10006, 2019.[7] Daniel Selsam, Matthew Lamm, Benedikt Bünz, Percy Liang, Leonardo de Moura, and David L.Dill. Learning a SAT solver from single-bit supervision. CoRR, abs/1802.03685, 2018.[8] Mingzhe Wang, Yihe Tang, Jian Wang, and Jia Deng. Premise selection for theorem proving bydeep graph embedding. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach,Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural InformationProcessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9December 2017, Long Beach, CA, USA, pages 2783–2793, 2017.[9] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. Acomprehensive survey on graph neural networks, 2019. cite arxiv:1901.00596Comment: updatedtables and references.Appendix A Project Plan: Bolzano-Weierstrass TheoremThe MPTP Challenge [4] consists of the Bolzano-Weierstrass theorem and its 252 auxiliarylemmas, constituting a relatively small, consistent problem domain. One part of the challengeis to prove the theorem and all lemmas from scratch using in each derivation only basic axioms,hence forcing long proofs. In this setup, we believe that curriculum learning can be very usefuland we intend to try to tackle the challenge with FLoP.3Update on FLoP Zombori, Csiszárik, Michalewski, Kaliszyk, UrbanAppendix B Project Plan: Backward Hindsight Experi-ence ReplayHindsight Experience Replay (HER) [1] is a clever approach to alleviate reward sparsity prob-lems in RL environments. Its core idea is to take an unsuccessful exploration trajectory, observestate S that it reached (as opposed to target state T) and then replay the same trajectory, whilepretending that the target state is now S. During the replay, the agent is rewarded for reachingthe new target.HER is directly applicable to a theorem proving environment that performs forward reasoning:each theorem proving attempt some valid consequences of the axioms, even if not the targetconjecture, so it makes sense to assume the new target in the replay. However, it is not obvioushow to do it for backward reasoning.The aim of our project is to redesign HER for the setting of a backward theorem prover.B.1 SetupWe want to train a backward theorem prover, i.e., one that starts from a target formula (goal)and at each inference step reduces the current goal to a list of other goals. Once a goal isidentical to some axiom or previously known lemma, the goal is closed and we can proceed totry to prove the remaining goals. The proof is complete when all goals have been closed.B.2 Core IdeaWe use Hindsight Experience Replay to provide denser reward to the guidance model. Considera single theorem proving attempt. If all goals are closed, then we have obtained a proof of thetarget and we can give positive reward to the policy. If there are some open goals, then wecan pretend that those goals were among the initial axioms and give positive reward in thismodified setting.B.3 ComponentsThe system has four components:1. Embedder e: takes a formula and maps it into a vector in Rn. This is most likely agraph neural network.2. Aggregator c: takes a set of formula (axiom) embeddings e(a1), e(a2) . . . e(ak) and mapsit into a single aggregate embedding, which represents the conjuction of the formulae. Thiscould be a recurrent neural network, though some permutation invariant solution wouldbe best.3. Policy p: takes a goal embedding and an aggregate axiom embedding and returnsan action probability distribution.4. Value v: takes a goal embedding and an aggregate axiom embedding and returnsa scalar value of the goal, given the axioms.4Update on FLoP Zombori, Csiszárik, Michalewski, Kaliszyk, UrbanThese components are trained together, end-to-end.B.4 TrainingWe iterate the steps below:1. Select a problem with goal g0 and axioms (lemmas) a1, a2, . . . ak2. Compute initial goal embedding eg0 = e(g0) and aggregate axiom embedding ea =c(e(a0), e(a1), . . . e(ak))3. Try to prove the goal based on the current policy p(egi , ea)4. Perform a gradient step based on the proving attempt for the value and policy, propagatingthe gradients all the way to the aggreagator and embedder as well.5. If the proof attempt failed, i.e., we are left with open goals og0, og1, . . . ogl, then(a) Compute new aggregate embeddingeâ = c(e(a0), e(a1), . . . e(ak), e(og0), e(og1), . . . e(ogl))(b) Replay the same inference steps with the new aggregate axiom embedding in thepolicy p(egi , eâ)(c) The open goals (ogi) are now axioms, so the proof is complete, hence we give positivereward and perform a gradient step.B.5 Benefits• We provide positive reward for every single theorem proving attempt.• The policy receives a representation of the axiom set (knowledge base) and can makemore informed decisions.• This works with any RL algorithm. There is no need of a DAGGER like setup, withseparate phases of data collection and supervised learning.B.6 Difficulties• Building a meaningful aggregate embedding of the available knowledge base (all axiomsand lemmas) might be hard and might be very slow. Some ideas to address this:– Use premise selection to restrict the aggregator to a handful of lemmas.– Precompute the aggregate embedding of all the lemmas and only ”incorporate” theembeddings of the axioms to the aggregate lemma embedding for each problem• Different open goals might be related due to sharing some variables. When we add theopen goals as new axioms in HER, we have to make sure that the axioms are consistent.E.g. when we have two open goals f(X) and ¬f(X), there is no way to add axioms thatsatisfy both.5

Update on FLoP, a Reinforcement Learning based Theorem Prover

http://real.mtak.hu/117261/1/paper_29.pdf

Update on FLoP, a Reinforcement Learning based Theorem Prover

Abstract

Similar works

Full text

Available Versions

Repository of the Academy's Library