Overfitting in Synthesis: Theory and Practice (Extended Version)

In syntax-guided synthesis (SyGuS), a synthesizer's goal is to automatically generate a program belonging to a grammar of possible implementations that meets a logical specification. We investigate a common limitation across state-of-the-art SyGuS tools that perform counterexample-guided inductive synthesis (CEGIS). We empirically observe that as the expressiveness of the provided grammar increases, the performance of these tools degrades significantly. We claim that this degradation is not only due to a larger search space, but also due to overfitting. We formally define this phenomenon and prove no-free-lunch theorems for SyGuS, which reveal a fundamental tradeoff between synthesizer performance and grammar expressiveness. A standard approach to mitigate overfitting in machine learning is to run multiple learners with varying expressiveness in parallel. We demonstrate that this insight can immediately benefit existing SyGuS tools. We also propose a novel single-threaded technique called hybrid enumeration that interleaves different grammars and outperforms the winner of the 2018 SyGuS competition (Inv track), solving more problems and achieving a $5\times$ mean speedup.Comment: 24 pages (5 pages of appendices), 7 figures, includes proofs of theorem

Synthesis of asynchronous distributed systems from global specifications

The synthesis problem asks whether there exists an implementation for a given formal specification and derives such an implementation if it exists. This approach enables engineers to think on a more abstract level about what a system should achieve instead of how it should accomplish its goal. The synthesis problem is often represented by a game between system players and environment players. Petri games define the synthesis problem for asynchronous distributed systems with causal memory. So far, decidability results for Petri games are mainly obtained for local winning conditions, which is limiting as global properties like mutual exclusion cannot be expressed. In this thesis, we make two contributions. First, we present decidability and undecidability results for Petri games with global winning conditions. The global safety winning condition of bad markings defines markings that the players have to avoid. We prove that the existence of a winning strategy for the system players in Petri games with a bounded number of system players, at most one environment player, and bad markings is decidable. The global liveness winning condition of good markings defines markings that the players have to reach. We prove that the existence of a winning strategy for the system players in Petri games with at least two system players, at least three environment players, and good markings is undecidable. Second, we present semi-decision procedures to find winning strategies for the system players in Petri games with global winning conditions and without restrictions on the distribution of players. The distributed nature of Petri games is employed by proposing encodings with true concurrency. We implement the semi-decision procedures in a corresponding tool.Das Syntheseproblem stellt die Frage, ob eine Implementierung f ¨ur eine Spezifikation existiert, und generiert eine solche Implementierung, falls sie existiert. Diese Vorgehensweise erlaubt es Programmierenden sich mehr darauf zu konzentrieren, was ein System erreichen soll, und weniger darauf, wie die Spezifikation erf ¨ ullt werden soll. Das Syntheseproblem wird oft als Spiel zwischen einem System- und einem Umgebungsspieler dargestellt. Petri-Spiele definieren das Syntheseproblem f ¨ur asynchrone verteilte Systeme mit kausalem Speicher. Bisher wurden Resultate bez¨uglich der Entscheidbarkeit von Petri-Spiele meist f ¨ur lokale Gewinnbedingungen gefunden. In dieser Arbeit pr¨asentieren wir zuerst Resultate bez¨uglich der Entscheidbarkeit und Unentscheidbarkeit von Petri-Spielen mit globalen Gewinnbedingungen. Wir beweisen, dass die Existenz einer gewinnenden Strategie f ¨ur die Systemspieler in Petri- Spielen mit einer beschr¨ankten Anzahl an Systemspielern, h¨ochstens einem Umgebungsspieler und schlechten Markierungen entscheidbar ist. Wir beweisen ebenfalls, dass die Existenz einer gewinnenden Strategie f ¨ur die Systemspieler in Petri-Spielen mit mindestens zwei Systemspielern, mindestens drei Umgebungsspielern und guten Markierungen unentscheidbar ist. Danach pr¨asentieren wir Semi-Entscheidungsprozeduren, um gewinnende Strategien f ¨ur die Systemspieler in Petri-Spielen mit globalen Gewinnbedingungen und ohne Restriktionen f ¨ur die Verteilung von Spielern zu finden. Wir benutzen die verteilte Natur von Petri-Spielen, indem wir Enkodierungen einf ¨uhren, die Nebenl¨aufigkeit ausnutzen. Die Semi-Entscheidungsprozeduren sind in einem entsprechenden Tool implementiert

Computer Aided Verification

This open access two-volume set LNCS 11561 and 11562 constitutes the refereed proceedings of the 31st International Conference on Computer Aided Verification, CAV 2019, held in New York City, USA, in July 2019. The 52 full papers presented together with 13 tool papers and 2 case studies, were carefully reviewed and selected from 258 submissions. The papers were organized in the following topical sections: Part I: automata and timed systems; security and hyperproperties; synthesis; model checking; cyber-physical systems and machine learning; probabilistic systems, runtime techniques; dynamical, hybrid, and reactive systems; Part II: logics, decision procedures; and solvers; numerical programs; verification; distributed systems and networks; verification and invariants; and concurrency