This report presents VLSAT-2 (an acronym for "Very Large Boolean SATisfiability problems"),the second part of a benchmark suite to be used in scientific experiments and softwarecompetitions addressing SAT-solving issues.VLSAT-2 contains 100 benchmarks (50 satisfiable and 50 unsatisfiable formulas)of increasing complexity, proposed in DIMACS CNF format undera permissive Creative Commons license.25% of these benchmarks have been used during the 2020 and 2021 editionsof the International SAT Competition.VLSAT-2 (acronyme anglais de "très grands problèmes de satisfaisabilité booléenne")est le second volet d'une suite de tests destinée aux expérimentations scientifiqueset aux compétitions de logiciels pour la résolution de problèmes SAT.VLSAT-2 contient 100 tests (50 formules satisfaisables et 50 insatisfaisables)de complexité croissante, fournis en format DIMACS CNF sous une licence Creative Commons permissive.25% de ces tests ont été utilisés lors des éditions 2020 et 2021 de la compétition internationale sur la résolution SAT

Bouvier, Pierre

Garavel, Hubert

INRIA a CCSD electronic archive server

ISSN0249-0803ISRNINRIA/RT--0514--FR+ENGTECHNICALREPORTN° 0514September 2021Project-Team CONVECSThe VLSAT-2Benchmark SuitePierre Bouvier, Hubert GaravelRESEARCH CENTREGRENOBLE – RHÔNE-ALPESInovallée655 avenue de l’Europe Montbonnot38334 Saint Ismier CedexThe VLSAT-2 Benchmark SuitePierre Bouvier, Hubert GaravelProject-Team CONVECSTechnical Report n° 0514 — September 2021 — 8 pagesAbstract: This report presents VLSAT-2 (an acronym for “Very Large Boolean SATisfiabilityproblems”), the second part of a benchmark suite to be used in scientific experiments and softwarecompetitions addressing SAT-solving issues. VLSAT-2 contains 100 benchmarks (50 satisfiableand 50 unsatisfiable formulas) of increasing complexity, proposed in DIMACS CNF format undera permissive Creative Commons license. 25% of these benchmarks have been used during the2020 and 2021 editions of the International SAT Competition.Key-words: benchmark suite, Boolean satisfiability problem, data set, DIMACS CNF, Nested-Unit Petri Net, NUPN, Petri Net, SAT formula, SAT solvingLe jeu de tests VLSAT-2Résumé : VLSAT-2 (acronyme anglais de “très grands problèmes de satisfaisabilité booléenne”)est le second volet d’une suite de tests destinée aux expérimentations scientifiques et aux compéti-tions de logiciels pour la résolution de problèmes SAT. VLSAT-2 contient 100 tests (50 formulessatisfaisables et 50 insatisfaisables) de complexité croissante, fournis en format DIMACS CNFsous une licence Creative Commons permissive. 25% de ces tests ont été utilisés lors des édi-tions 2020 et 2021 de la compétition internationale sur la résolution SAT.Mots-clés : DIMACS CNF, ensemble de données, formule SAT, Nested-Unit Petri Net, NUPN,problème SAT, réseau de Petri, satisfaisabilité booléenne, suite de testsThe VLSAT-2 Benchmark Suite 31 Benchmark DescriptionVLSAT-21 is a collection of 100 SAT formulas. Many of these formulas are difficult to handle bycurrent SAT solvers. One half of these formulas is satisfiable, while the other half is not.Each formula is provided as a separate file, expressed in Conjunctive Normal Form and encodedin the DIMACS CNF format2. Each file is then compressed using bzip2 to save disk space andallow faster downloads. The 100 formulas require 5.2 gigabytes of disk space and 1.4 gigabyteswhen compressed using bzip2.The VLSAT-2 benchmarks are licensed under the CC-BY Creative Commons Attribution 4.0International License3.25% of the VLSAT-2 benchmarks have been selected by the organizers of recent SAT Competi-tions: 7 satisfiable and 7 unsatisfiable formulas have been chosen for the SAT Competition 2020,and 5 satisfiable and 8 unsatisfiable formulas have been chosen for the SAT Competition 2021 [2].2 Scientific ContextInteresting Boolean formulas can be generated as a by-product of our recent work [3] on thedecomposition of Petri nets into networks of automata, a problem that has been around sincethe early 70s. Concretely, we developed a tool chain that takes as input a Petri net (which mustbe ordinary, safe, and hopefully not too large) and produces as output a network of automatathat execute concurrently and synchronize using shared transitions. Precisely, this networkis expressed as a Nested-Unit Petri Net (NUPN) [4], i.e., an extension of a Petri net, in whichplaces are grouped into sets (called units) that denote sequential components. A NUPN providesa proper structuring of its underlying Petri net, and enables formal verification tools to be moreefficient in terms of memory and CPU time. Hence, the NUPN concept has been implementedin many tools and adopted by software competitions, such as the Model Checking Contest4 [8, 7]and the Rigorous Examination of Reactive Systems challenge5 [5, 9, 6]. Each NUPN generatedby our tool chain is flat, meaning that its units are not recursively nested in each other, andunit-safe, meaning that each unit has at most one execution token at a time.Our tool chain works by reformulating concurrency constraints on Petri nets as logical problems,which can be later solved using third-party software, such as SAT solvers, SMT solvers, and toolsfor graph coloring and finding maximum cliques [3]. We applied our approach to a large collectionof more than 12,000 Petri nets from multiple sources, many of which are related to industrialproblems, such as communication protocols, distributed systems, and hardware circuits. We thusgenerated a huge collection of Boolean formulas, from which we carefully selected a subset offormulas matching the requirements of the SAT Competition.3 Structure of FormulasEach of our formulas was produced for a particular Petri net. A formula depends on three factors:1https://cadp.inria.fr/resources/vlsat/2.html2http://www.satcompetition.org/2009/format-benchmarks2009.html3License terms available from http://creativecommons.org/licenses/by/4.04https://mcc.lip6.fr5http://rers-challenge.orgRT n° 05144 P. Bouvier & H. Garavel• the set P of the places of the Petri net;• a concurrency relation ‖ defined over P , such that p ‖ p′ iff both places p and p′ maysimultaneously have an execution token; and• a chosen number n of units.A formula expresses whether there exists a partition of P into n subsets Pi (1 ≤ i ≤ n) suchthat, for each i, and for any two places p and p′ of Pi, p 6= p′ =⇒ ¬ (p ‖ p′). A model of thisformula is thus an allocation of places into n units, i.e., a valid decomposition of the Petri net.This can also be seen as an instance of the graph coloring problem, in which n colors are to beused for the graph with vertices defined by the places of P and edges defined by the concurrencyrelation. A formula is only satisfiable if the value of n is large enough (namely, greater than orequal to the chromatic number of the graph), so that at least one decomposition exists.More precisely, each formula was generated as follows. For each place p and each unit u, wecreated a propositional variable xpu that is true iff place p belongs to unit u. We then addedconstraints over these variables:• For each unit u and each two places p and p′ such that p ‖ p′ and #p < #p′, where#p is a bijection from places names to the interval [1, card (P )], we added the constraint¬xpu ∨ ¬xp′u to express that two concurrent places cannot be in the same unit.• For each place p, we could have added the constraint∨u xpu to express that p belongsto at least one unit, but this constraint was too loose and allowed n! similar solutions,just by permuting unit names. We thus replaced this constraint by a stricter one thatbreaks the symmetry between units: for each place p, we added the refined constraint∨1≤#u≤min(#p,n) xpu, where #u is a bijection from unit names to the interval [1, n].Figure 1 illustrates, for a typical Petri net, how the resolution time evolves as the chosen numberof units increases. If the value of n is small (resp. large) enough, it is relatively easy to prove thatthe formula is satisfiable (resp. unsatisfiable) because there are not enough (resp. too many)colors. The difficulty comes when n gets close to the chromatic number (which is equal to 13on Fig. 1), as the number of combinations to be examined for proving unsatisfiability explodes,despite the introduction of symmetry-breaking constraints, which make unsatisfiable formulasmuch easier and satisfiable formulas slightly harder.4 Selection of BenchmarksUsing the approach presented in Sections 2 and 3, we previously published a test suite, namedVLSAT-1 [1], of 100 formulas. However, VLSAT-1 only contains satisfiable formulas, as it wasdesigned for the Model Counting Competition, which seeks formulas accepting a large number ofmodels. For the SAT Competition, we therefore undertook the production of a different collectioncontaining both satisfiable and unsatisfiable formulas, depending on the number of units chosenfor a given Petri net.We selected 50 satisfiable and 50 unsatisfiable formulas, carefully chosen among a large collectionof more than 132,000 formulas generated by our tool chain. We used five SAT solvers (namely,CaDiCal 1.3.0, Kissat 1.0.3, MathSAT 5.6.5, MiniSAT 2.2.0, and Z3 4.8.9) to reject all formulasthat can be solved in less than one minute of CPU time by at least one of these solvers, andthat can be solved within two hours by each of these solvers (experiments done on a Xeon E5-2630 v4 machine with 256 gigabytes of RAM). We also tried to provide formulas of increasingInriaThe VLSAT-2 Benchmark Suite 5complexities, with a good compromise between the size of a formula and the time taken by thefastest solver to process this formula.0 2 4 6 8 10 12 14 16 18 20 22 24 260.010.11101001000Number of unitsResolution time, in seconds (logarithmic scale)UNSAT SATFigure 1: Resolution times for a typical NUPN5E+2 2E+3 5E+3 2E+4 5E+48E+33E+48E+43E+58E+53E+68E+6satunsatNumber of variablesNumber of clausesFigure 2: Dispersion of the VLSAT-2 benchmarksRT n° 05146 P. Bouvier & H. Garavelvariables clauses type difficulty544 8738 UNSAT 3600 11,440 UNSAT 3684 9417 UNSAT 3684 13,953 UNSAT 2708 10,259 UNSAT 3736 11,022 UNSAT 2798 17,543 UNSAT 31000 22,250 SAT 41022 14,955 UNSAT 31125 15,795 UNSAT 31134 26,703 UNSAT* 41155 42,917 UNSAT* 41183 26,975 UNSAT 31209 29,889 UNSAT 31326 35,956 UNSAT 31352 42,432 UNSAT 31560 54,564 UNSAT 31586 57,751 UNSAT 32231 68,844 UNSAT 22340 70,758 UNSAT 22403 100,259 UNSAT 22548 119,204 UNSAT 33252 372,331 UNSAT 33456 192,912 SAT 23480 190,496 UNSAT 34424 545,056 UNSAT* 35152 824,642 UNSAT* 35525 327,765 UNSAT 35568 1,124,240 UNSAT 35600 1,042,700 UNSAT* 3452 334,035 SAT 311,130 1,186,888 SAT* 0 (148 s)11,280 4,223,777 UNSAT* 311,374 1,150,943 SAT* 1 (1802 s)11,664 5,532,624 UNSAT 412,690 1,481,522 SAT 214,016 1,374,747 SAT 414,280 6,781,327 UNSAT* 314,424 7,585,190 UNSAT* 414,637 1,778,453 SAT 214,640 1,323,246 UNSAT 515,249 1,937,993 SAT 215,440 1,409,906 UNSAT+ 515,704 1,804,650 SAT 315,960 1,464,039 UNSAT+ 516,269 2,203,672 SAT 316,297 1,562,268 UNSAT 516,676 1,598,591 SAT+ 216,788 9,021,307 UNSAT 317,688 1,741,702 UNSAT 5variables clauses type difficulty18,250 2,995,915 SAT 219,565 3,665,001 SAT 220,424 2,157,568 UNSAT 521,114 2,240,429 UNSAT+ 521,190 2,597,791 SAT 321,546 2,615,351 SAT 321,573 2,289,124 SAT 322,032 3,022,731 SAT 223,961 2,714,844 UNSAT 524,450 2,770,239 SAT+ 1 (721 s)26,104 3,131,630 UNSAT 526,606 3,191,844 SAT 226,988 3,251,923 UNSAT 527,507 3,314,450 SAT 328,930 6,497,511 SAT 429,040 3,874,024 SAT 229,205 3,712,921 UNSAT 529,456 6,615,638 SAT 229,736 3,780,419 SAT* 229,945 3,796,274 SAT 430,195 3,855,554 UNSAT+ 530,744 3,925,645 SAT+ 531,552 5,400,750 SAT 432,480 4,362,044 UNSAT+ 533,040 4,437,242 SAT 433,582 4,529,625 UNSAT 534,099 4,695,729 SAT 334,161 4,607,712 SAT+ 435,929 5,082,743 UNSAT+ 536,518 5,166,057 SAT 436,603 5,137,412 SAT 336,792 5,273,558 SAT 237,758 5,364,539 SAT+* 439,552 5,878,762 UNSAT+ 540,170 5,970,608 SAT+ 540,896 6,104,639 UNSAT 541,535 6,200,014 SAT 441,875 6,420,498 SAT 445,150 7,165,285 SAT 446,440 7,369,989 SAT 247,817 1,337,056 SAT 257,038 10,572,502 SAT+ 558,380 10,775,711 SAT 459,204 10,973,962 SAT 467,996 13,708,722 SAT 568,760 13,862,744 SAT 569,524 14,016,766 SAT 570,288 14,170,788 SAT* 471,816 14,478,832 SAT 383,334 20,350,783 SAT 4Table 1: List of VLSAT-2 formulasInriaThe VLSAT-2 Benchmark Suite 7The VLSAT-2 formulas are listed in Table 1. Those marked with a plus (resp. a star) in thetable have been selected by the organizers of the SAT Competition 2020 (resp. 2021). Thecolumn “difficulty” contains a number from 0 (easy) to 5 (hard) indicating how many of the fiveaforementioned SAT solvers failed to solve the corresponding formula within two hours. For theeasy values 0 and 1, the average number of seconds taken by the tools that managed to solve theformula is given.Figure 2 shows the dispersion of the VLSAT-2 benchmarks for both satisfiable and unsatisfiableformulas. In general, and as confirmed by Fig. 1, satisfiable formulas need to be much larger (inthe number of variables and clauses) than unsatisfiable ones to reach the same level of difficulty.AcknowledgementsThe experiments presented in this paper were carried out using the Grid’50006 testbed, sup-ported by a scientific interest group hosted by Inria and including Cnrs, Renater and severalUniversities as well as other organizations.References[1] Pierre Bouvier and Hubert Garavel. The VLSAT-1 Benchmark Suite. Technical Report RT-0510,INRIA, Grenoble, France, November 2020. Available from https://hal.inria.fr/hal-03007233and https://arxiv.org/abs/2011.11049.[2] Pierre Bouvier and Hubert Garavel. SAT-Competition Benchmarks Spawning from ConcurrencyTheory. In Tomáš Balyo, Nils Froleyks, Marijn J. H. Heule, Markus Iser, Matti Järvisalo, andMartin Suda, editors, Proceedings of SAT Competition 2021 – Solver and Benchmark Descriptions,pages 47–48. Report B-2021-1, University of Helsinki, Department of Computer Science, July 2021.[3] Pierre Bouvier, Hubert Garavel, and Hernán Ponce de León. Automatic Decomposition of Petri Netsinto Automata Networks – A Synthetic Account. In Ryszard Janicki, Natalia Sidorova, and ThomasChatain, editors, Proceedings of the 41st International Conference on Application and Theory ofPetri Nets and Concurrency (PETRI NETS’20), Paris, France, volume 12152 of Lecture Notes inComputer Science, pages 3–23. Springer, June 2020.[4] Hubert Garavel. Nested-Unit Petri Nets. Journal of Logical and Algebraic Methods in Programming,104:60–85, April 2019.[5] Marc Jasper, Maximilian Fecke, Bernhard Steffen, Markus Schordan, Jeroen Meijer, Jaco van de Pol,Falk Howar, and Stephen F. Siegel. The RERS 2017 Challenge and Workshop. In Hakan Erdogmusand Klaus Havelund, editors, Proceedings of the 24th ACM SIGSOFT International SPIN Symposiumon Model Checking of Software (SPIN’17), Santa Barbara, CA, USA, pages 11–20. ACM, July 2017.[6] Marc Jasper, Malte Mues, Alnis Murtovi, Maximilian Schlüter, Falk Howar, Bernhard Steffen,Markus Schordan, Dennis Hendriks, Ramon R. H. Schiffelers, Harco Kuppens, and Frits W. Vaan-drager. RERS 2019: Combining Synthesis with Real-World Models. In Dirk Beyer, Marieke Huis-man, Fabrice Kordon, and Bernhard Steffen, editors, Proceedings of the 25th International Confer-ence on Tools and Algorithms for the Construction and Analysis of Systems (TACAS’19), Part III:TOOLympics, Prague, Czech Republic, pages 101–115. Springer, April 2019.[7] Fabrice Kordon, Hubert Garavel, Lom Messan Hillah, Emmanuel Paviot-Adet, Loïg Jezequel, Fran-cis Hulin-Hubard, Elvio Amparore, Marco Beccuti, Bernard Berthomieu, Hugues Evrard, Peter G.Jensen, Didier Le Botlan, Torsten Liebke, Jeroen Meijer, Jiří Srba, Yann Thierry-Mieg, Jaco van dePol, and Karsten Wolf. MCC’2017 – The Seventh Model Checking Contest. Transactions on PetriNets and Other Models of Concurrency, XIII:181–209, 2018.6https://www.grid5000.frRT n° 05148 P. Bouvier & H. Garavel[8] Fabrice Kordon, Hubert Garavel, Lom Messan Hillah, Emmanuel Paviot-Adet, Loïg Jezequel, CésarRodríguez, and Francis Hulin-Hubard. MCC’2015 – The Fifth Model Checking Contest. Transactionson Petri Nets and Other Models of Concurrency, XI:262–273, 2016.[9] Bernhard Steffen, Marc Jasper, Jeroen Meijer, and Jaco van de Pol. Property-Preserving Generationof Tailored Benchmark Petri Nets. In Proceedings of the 17th International Conference on Applicationof Concurrency to System Design (ACSD’17), Zaragoza, Spain, pages 1–8. IEEE Computer Society,June 2017.InriaRESEARCH CENTREGRENOBLE – RHÔNE-ALPESInovallée655 avenue de l’Europe Montbonnot38334 Saint Ismier CedexPublisherInriaDomaine de Voluceau - RocquencourtBP 105 - 78153 Le Chesnay Cedexinria.frISSN 0249-0803

Le jeu de tests VLSAT-2

https://hal.inria.fr/hal-03337115/file/RT-0514.pdf

Le jeu de tests VLSAT-2

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server