We describe an integration of program synthesis into Uclid5, a formal
modelling and verification tool. To the best of our knowledge, the new version
of Uclid5 is the only tool that supports program synthesis with bounded model
checking, k-induction, sequential program verification, and hyperproperty
verification. We use the integration to generate 25 program synthesis
benchmarks with simple, known solutions that are out of reach of current
synthesis engines, and we release the benchmarks to the community

Cheang, Kevin

Mora, Federico

Polgreen, Elizabeth

Seshia, Sanjit A.

English

arXiv

Seshia, Sanjit A

eScholarship - University of California

UC BerkeleyUC Berkeley Previously Published WorksTitleSynthesis in Uclid5Permalinkhttps://escholarship.org/uc/item/63k829jcAuthorsMora, FedericoCheang, KevinPolgreen, Elizabethet al.Publication Date2020-07-13 Peer reviewedeScholarship.org Powered by the California Digital LibraryUniversity of CaliforniaarXiv:2007.06760v2  [cs.PL]  17 Jul 2020Synthesis in Uclid5Federico MoraUniversity of California, BerkeleyKevin CheangUniversity of California, BerkeleyElizabeth PolgreenUniversity of California, BerkeleySanjit A. SeshiaUniversity of California, BerkeleyAbstractWedescribe an integration of program synthesis intoUclid5,a formal modelling and verification tool. To the best of ourknowledge, the new version of Uclid5 is the only tool thatsupports program synthesis with bounded model checking,k-induction, sequential programverification, and hyperprop-erty verification. We use the integration to generate 25 pro-gram synthesis benchmarks with simple, known solutionsthat are out of reach of current synthesis engines, and werelease the benchmarks to the community.1 IntroductionFormal verification can be a time-consuming task that re-quires significant manual effort. Especially for complex sys-tems, users often need to manually provide, for example,loop invariants, function summaries, or environment mod-els. Synthesis has the potential to alleviate some of this man-ual burden [16]. For example, prior work has used synthesisto reason about program loops [5], and to automate programrepair [9].We believe this is a promising direction, but, for itto make a real impact, verification tools need to offer flexiblesynthesis integration, generic support for proof procedures,and a capable synthesis engine back-end.In this work, we primarily address the first two require-ments. Specifically, we integrate program synthesis into theUclid5 [14] formal modelling and verification tool by allow-ing users to declare functions to synthesize and to use thesefunctions freely.WhileUclid5has previously supported useof synthesis, it only supported invariant synthesis througha special command that was independent of verification. Weuse the new synthesis integration to generate 25 benchmarksfrom existing verification tasks. These benchmarkshave smallsolutions, but are out of reach for current synthesis engines.We hope that they will help the synthesis engine develop-ment effort, particularly for syntax-guided synthesis [1].Illustrative Example. Consider theUclid5model in Fig. 1,which represents a Fibonacci sequence. The (hypothetical)user wants to prove by induction that the invariant a_le_bat line 13 always holds. Unfortunately, the proof fails be-cause the invariant is not inductive. Without synthesis, theuser would need to manually strengthen the invariant untilit became inductive. However, the user can ask Uclid5 toautomatically do this for them. Fig. 1 demonstrates this onlines 16, 17 and 18. Specifically, the user specifies a functionto synthesize called h at lines 16 and 17, and then uses h atline 18 to strengthen the existing set of invariants. Giventhis input, Uclid5, using e.g. cvc4 [2] as a synthesis engine,will automatically generate the function h(x, y) = x >=0, which completes the inductive proof.In this example, the function to synthesize represents aninductive invariant. However, functions to synthesize aretreated exactly like any interpreted function in Uclid5: theuser could have called h anywhere in the code. Furthermore,this example uses induction and a global invariant, however,the user could also have used a linear temporal logic (LTL)specification and bounded model checking (BMC). In thissense, our integration is fully flexible and generic.Contributions. We present an integration of synthesisinto the verification tool Uclid5, allowing users to generateprogram synthesis queries for unknown parts of a systemthey wish to verify. The integration is a natural extension ofthe existing Uclid5 language, and to the best of our knowl-edge, is the first to support program synthesis with boundedmodel checking, k-induction, sequential program verifica-tion, and hyperproperty verification. The synthesis queriesUclid5 generates are in the standard sygus-if [12] specifi-cation language. We use this tool to generate a 25 sygus-ifsynthesis benchmarks from existing verification queries andrelease these benchmarks to the community.2 Related workProgram sketching [17] synthesizes expressions to fill holesin programs, and has subsequently been applied to programrepair [8, 9]. Uclid5 aims to be more flexible than this work,allowing users to declare unknown functions even in theverification annotations, as well as supporting multiple ver-ification algorithms and types of properties. Rosette [18]provides support for synthesis and verification, but the syn-thesis is limited to bounded specifications of sequential pro-grams, whereas Uclid5 can also synthesize programs thatsatisfy unbounded specifications, by using proof procedureslike induction. Formal synthesis algorithms have been usedto assist in verification tasks, such as safety and termina-tion of loops [5], and generating invariants [7, 19], but noneof this work to-date integrates program synthesis fully intoan existing verification tool. Before this new synthesis inte-gration, Uclid5 supported synthesis of inductive invariants.The key insight of this work is to generalize the synthesisFederico Mora, Kevin Cheang, Elizabeth Polgreen, and Sanjit A. Seshia1 module main {2 // Part 1: System Description .3 var a, b : integer;4 init {5 a, b = 0, 1;6 }7 next {8 a', b' = b, a + b;9 }1011 // Part 2: System Specification .12 invariant a_le_b: a <= b;1314 // Part 3: (NEW) Synthesis Integration15 synthesis function16 h(x : integer , y : integer): boolean;17 invariant hole: h(a, b);1819 // Part 4: Proof Script.20 control {21 induction ;22 check;23 print_results ;24 }25 }Figure 1. Uclid5 Fibonacci model. Part 3 shows the newsynthesis syntax, and how to find an auxiliary invariant.support, and to unify all synthesis tasks in Uclid5 by re-using the verification back-end.3 From Verification to Program SynthesisIn this section, we give the necessary background on pro-gram synthesis, and the existing verification techniques in-side of Uclid5. We then describe how we combine the twoto realize synthesis in Uclid5.3.1 Program SynthesisThe program synthesis problem corresponds to the second-order query∃f ∀®xσ (f , ®x),where f is the function to synthesize, ®x is the set of all pos-sible inputs, and σ is the specification to be satisfied.3.2 Verification in Uclid5At at high level, Uclid5 takes in a model, generates a set ofverification conditions, asks a satisfiability modulo theory(SMT) solver [4] to check the verification conditions, andthen returns the results to the user. This process is the sameregardless of the proof procedure used. The important point,is that Uclid5 encodes the violation of each independentverification condition as a separate smt-lib query.Let Pi (®x) encode the ith verification condition, and takethe ith smt-lib query to be checking the validity of∃®x ¬Pi (®x),where Pi contains no free variables. We say that there isfront-endSymbolic Simulatorsynth-lib interfacesmt-lib interface sygus-if interfaceFigure 2. Overview of synthesis in Uclid5. Dashed blocksindicate blocks introduced for the new synthesis integra-tion.a counterexample to the ith verification query if the query∃®x ¬Pi (®x) is valid. Verification of amodel withn verificationconditions succeeds iff there are no counter-examples:∀®xi=n∧i=0Pi (®x).3.3 Synthesis Encoding in Uclid5Given aUclid5model in which the user has declared a func-tion to synthesize, f , we wish to construct a synthesis querythat is satisfied iff there is an f for which all the verificationconditions pass for all possible inputs. We build this synthe-sis query by taking the conjunction of the negation of all theverification queries. Specifically, we check the validity of∃f ∀®xi=n∧i=0Pi (f , ®x)where each Pi encodes a verification condition that may re-fer to the function to synthesize, f . Note the similarity be-tween this query and the standard program synthesis formu-lation: the specification for synthesis, σ , from the equationin Sec. 3.1, is now replaced with the conjunction of all theverification conditions.With this observation, to enable syn-thesis for any verification procedure in Uclid5, all we do islet users declare and use functions to synthesize.4 ImplementationThe Uclid5 verification tool is constructed as shown in Fig-ure 2. An input Uclid5 model is parsed by the front-endinto an abstract syntax tree. From this abstract syntax tree,a symbolic simulator generates an assertion stack that con-tains an assertion for each verification condition. Prior toour work, assertions were then passed to an smt-lib inter-face which converted the assertions to smt-lib and calleda solver. The new Uclid5 instead uses a new intermediaterepresentation, synth-lib, that is easily passed to either anSMT solver or a synthesis engine. This architecture allowsus to use the same code that generates verification queriesfor synthesis.Synthesis in Uclid51 (synth -blocking -fun h ((x Int) (y Int)) Bool )2 ;(define -fun h ((x Int) (y Int)) Bool (>= x 0))3 (declare -fun initial_b () Int)4 (declare -fun initial_a () Int)5 (declare -fun new_a () Int)6 (declare -fun new_b () Int)7 (assert (or8 (not (and (<= initial_a initial_b ) (h 0 1)))9 (and10 (and (<= initial_a initial_b ) (h initial_ainitial_b ))11 (= new_a initial_b )12 (= new_b (+ initial_a initial_b ))13 (not (and (<= new_a new_b) (h new_a new_b))))))14 (check -sat)Figure 3. synth-lib induction query of Fig. 1The synth-lib representation is smt-lib [3], but with oneextra command borrowed from sygus-if [12]. The syntaxfor the new command is(synth-blocking-fun 〈fname〉((〈argname〉 〈argsort〉)∗) 〈rsort〉 〈grammar〉?),where 〈fname〉 is the name of the function, 〈argname〉 is thename of an argument, 〈argsort〉 is the sort of the correspond-ing argument, there are zero or more arguments, 〈rsort〉is the sort returned by the function, and 〈grammar〉 is anoptional syntactic specification for the function body. Intu-itively, a synth-lib query with a single synth-blocking-fun declaration asks “is there a function that makes thisunderlying smt-lib query unsatisfiable?”Fig. 3 shows the synth-lib query corresponding to theFibonacci model in Fig. 1. A synthesis engine might solvethe query in Fig. 3 by finding the function h(x, y) = x>= 0. This is a correct solution because the correspondingsmt-lib query—which we can get by commenting out line1 of Fig. 3 and uncommenting line 2—is unsatisfiable.The semantics of synth-lib is exactly that of smt-libwhen no function to synthesize is on the assertion stack, andassertions are passed directly to the SMT solver. When theassertion stack contains a function to synthesize, Uclid5 ap-plies the following four rewrite rules to convert synth-libinto sygus-if:1. (assert a) → (constraint (not a))2. (declare-fun a (s0...sn−1) sn → (declare-var a s0 ->...-> sn )3. synth-blocking-fun→ synth-fun4. check-sat→ check-synthThe first rewrite rule is the most important: it implementsthe following equivalence∃f ¬∃®xi=n∨i=0¬Pi (f , ®x) ≡ ∃f ∀®xi=n∧i=0Pi (f , ®x),where the left hand side is the form of queries in synth-lib,and the right hand side is the corresponding query in sygus-if. The source code for Uclid5 is available online [15].5 Benchmark SuiteThe integration of synthesis into Uclid5 allows us to gen-erate synthesis benchmarks from any Uclid5 verificationtask. We thus present a set of 25 benchmarks with known,small solutions that are out of reach of existing synthesissolvers. These benchmarks use induction, BMC, LTL speci-fications, and sequential code. To conform to the sygus-iflanguage, we limited ourselves to bit-vector, integer, array,and boolean data-types, and did not use verification tasksthat required quantifiers. All benchmarks are available on-line [11].The benchmarks come from four different sources. Fourbenchmarks come from a simplified model of the Two PhaseCommit protocol, written in P [6]; three benchmarks comefrom Sahai at al’s [13] work on hyperproperty verification;six benchmarks come from Uclid5’s documentation; andthe remaining 12 benchmarks come frommodels used in UCBerkeley’s EECS 219C course. In all cases, we constructedthe benchmarks by replacing small parts of either auxiliaryinvariants or parts of existing codewith functions to synthe-size. 12 benchmarks come from models that use induction,and 13 from models that use LTL specifications and BMC.All 25 benchmarks are difficult for existing state-of-the-artengines, but are a reasonable target for synthesis engines.6 Conclusions and Future WorkWehave presented an integration of synthesis into theUclid5verification tool, allowing users to generate synthesis queriesfor unknown parts of a system they wish to verify. This in-tegration is compatible with all verification algorithms cur-rently supported byUclid5, and generates synthesis queriesin the standard sygus-if format.In the future, we intend to apply synthesis in Uclid5 tothe verification of distributed systems written in P. Priorwork has been successfully in finding invariants for boundeddistributed systems, and then generalizing the invariants tothe unbounded setting [10]. We plan to explore these ap-proaches with Uclid5 now that we can easily switch be-tween synthesis using e.g. BMC and k-induction.AcknowledgmentsThis work was supported in part by NSF grants 1739816 and1837132, a gift from Intel under the SCAP program, SRCTask 2867.001, and the iCyPhy center.Federico Mora, Kevin Cheang, Elizabeth Polgreen, and Sanjit A. SeshiaReferences[1] Rajeev Alur, Rastislav Bodik, Garvit Juniwal, Milo M. K. Martin,Mukund Raghothaman, Sanjit A. Seshia, Rishabh Singh, ArmandoSolar-Lezama, Emina Torlak, and Abhishek Udupa. 2013. Syntax-Guided Synthesis. In FMCAD. 1–17.[2] Clark Barrett, Christopher L. Conway, Morgan Deters, LianaHadarean, Dejan Jovanovic, Tim King, Andrew Reynolds, and CesareTinelli. 2011. CVC4. In CAV. 171–177.[3] Clark Barrett, Pascal Fontaine, and Cesare Tinelli. 2016. The Satisfia-bility Modulo Theories Library (SMT-LIB). www.SMT-LIB.org.[4] Clark Barrett, Roberto Sebastiani, Sanjit A. Seshia, and Cesare Tinelli.2009. Satisfiability Modulo Theories. In Handbook of Satisfiability,Armin Biere, Hans van Maaren, and Toby Walsh (Eds.). Vol. 4. IOSPress, Chapter 8.[5] Cristina David, Daniel Kroening, and Matt Lewis. 2015. Using Pro-gram Synthesis for Program Analysis. In LPAR. Springer, 483–498.[6] Ankush Desai, Vivek Gupta, Ethan Jackson, Shaz Qadeer, Sriram Raja-mani, and Damien Zufferey. 2013. P: safe asynchronous event-drivenprogramming. PLDI (2013), 321–332.[7] Grigory Fedyukovich and Rastislav Bodík. 2018. Accelerating Syntax-Guided Invariant Synthesis. In TACAS (1). Springer, 251–269.[8] Jinru Hua, Mengshi Zhang, Kaiyuan Wang, and Sarfraz Khurshid.2018. Towards practical program repair with on-demand candidategeneration. In ICSE. ACM, 12–23.[9] Xuan-Bach D. Le, Duc-Hiep Chu, David Lo, Claire Le Goues, andWillemVisser. 2017. S3: syntax- and semantic-guided repair synthesisvia programming by examples. In ESEC/SIGSOFT FSE. ACM, 593–604.[10] Haojun Ma, Aman Goel, Jean-Baptiste Jeannin, Manos Kapritsos,Baris Kasikci, and Karem A Sakallah. 2019. I4: incremental infer-ence of inductive invariants for verification of distributed protocols.In OSDI. ACM, 370–384.[11] Federico Mora. 2020. UCLID5 Synthesis Benchmarks.hps://github.com/FedericoAureliano/synthesis-benchmarks[12] Mukund Raghothaman and Abhishek Udupa. 2014. Lan-guage to Specify Syntax-Guided Synthesis Problems.https://sygus.org/assets/pdf/SyGuS-IF.pdf.[13] Shubham Sahai, Rohit Sinha, and Pramod Subramanyan. 2020. Ver-ification of Quantitative Hyperproperties Using Trace EnumerationRelations. In CAV.[14] Sanjit Seshia and Pramod Subramanyan. 2018. UCLID5: Integratingmodeling, verification, synthesis, and learning. InMEMOCODE.[15] Sanjit Seshia and Pramod Subramanyan. 2020. UCLID5: A systemfor modeling, verification, and synthesis of computational systems.hps://github.com/uclid-org/uclid[16] Sanjit A. Seshia. [n.d.]. Combining Induction, Deduction, and Struc-ture for Verification and Synthesis. IEEE.[17] Armando Solar-Lezama. 2009. The sketching approach to programsynthesis. In Asian Symposium on Programming Languages and Sys-tems. Springer, 4–13.[18] Emina Torlak and Rastislav Bodík. 2013. Growing solver-aided lan-guages with rosette. In Onward! ACM, 135–152.[19] Hongce Zhang, Weikun Yang, Grigory Fedyukovich, Aarti Gupta, andSharad Malik. 2020. Synthesizing Environment Invariants for Modu-lar Hardware Verification. In VMCAI. Springer, 202–225.

Synthesis in Uclid5

Abstract

Similar works

Full text

Available Versions

eScholarship - University of California