Automatically Identifying Soundness and Completeness Errors in Program Analysis Tools

Program analysis tools (such as static analyzers, SMT solvers, and program verifiers) are extremely important for ensuring the correctness of a large variety of software systems. Very often, these tools are assumed to be sound (i.e., do not miss errors) and complete (i.e., have a low rate of false positives), otherwise their results are not reliable. However, these assumptions do not always hold in practice. Even if their theoretical designs have been proven correct, the actual implementations can still contain issues. We thus propose through this dissertation systematic techniques for automatically identifying soundness and completeness errors in the implementations of program analysis tools. Other types of bugs, such as performance or convergence issues, can be also detected as by-products. Our first contribution is a novel combination of automatic test case generation approaches for identifying soundness, precision, and termination issues in the implementations of numerical abstract domains, the main components of static analyzers based on abstract interpretation. We show that our technique effectively detects errors in widely-used libraries for numerical analyses, outperforming dynamic symbolic execution and grey-box fuzzing. Our work applies also to abstract domains that rely on machine learning to improve the performance of the analysis. Our second contribution is an automated approach for synthesizing SMT formulas that are satisfiable or unsatisfiable by construction. Together with the known ground truth, these are used to test the implementations of SMT solvers. We generate satisfiable formulas together with models, and unsatisfiable formulas together with unsat cores; being incrementally complex, they facilitate debugging and faster error localization. We evaluated our work on three widely-used SMT solvers, Z3-seq, Z3str3, and CVC4 and on the automata-based solver MT-ABC. Our experimental results show that our approach effectively detects soundness, performance, completeness, and precision problems. It is applicable also to MAX-SMT solvers. Our third contribution is an automated technique that allows the developers to detect completeness issues in SMT-based program verifiers and soundness errors in their axiomatizations. Moreover, our approach helps them devise better triggering strategies for all future runs of their tool with E-matching. We developed a novel algorithm for synthesizing the triggering terms necessary to complete unsatisfiability proofs using E-matching.We evaluated our work on benchmarks with known triggering issues from four program verifiers. Our experiments show that it successfully synthesized the missing triggering terms in the majority of the cases, and can significantly reduce the human effort in localizing and fixing the errors

Automatically Testing String Solvers

SMT solvers are at the basis of many applications, such as program verification, program synthesis, and test case generation. For all these applications to provide reliable results, SMT solvers must answer queries correctly. However, since they are complex, highly-optimized software systems, ensuring their correctness is challenging. In particular, state-of-the-art testing techniques do not reliably detect when an SMT solver is unsound. In this paper, we present an automatic approach for generating test cases that reveal soundness errors in the implementations of string solvers, as well as potential completeness and performance issues. We synthesize input formulas that are satisfiable or unsatisfiable by construction and use this ground truth as test oracle. We automatically apply satisfiability-preserving transformations to generate increasingly-complex formulas, which allows us to detect many errors with simple inputs and, thus, facilitates debugging. The experimental evaluation shows that our technique effectively reveals bugs in the implementation of widely-used SMT solvers and applies also to other types of solvers, such as model-counting constraint solvers. We focus on strings here, but our approach carries over to other theories and their combinations

Automatically testing string solvers

SMT solvers are at the basis of many applications, such as program verification, program synthesis, and test case generation. For all these applications to provide reliable results, SMT solvers must answer queries correctly. However, since they are complex, highly-optimized software systems, ensuring their correctness is challenging. In particular, state-of-the-art testing techniques do not reliably detect when an SMT solver is unsound. In this paper, we present an automatic approach for generating test cases that reveal soundness errors in the implementations of string solvers, as well as potential completeness and performance issues. We synthesize input formulas that are satisfiable or unsatisfiable by construction and use this ground truth as test oracle. We automatically apply satisfiability-preserving transformations to generate increasingly-complex formulas, which allows us to detect many errors with simple inputs and, thus, facilitates debugging. The experimental evaluation shows that our technique effectively reveals bugs in the implementation of widely-used SMT solvers and applies also to other types of solvers, such as automata-based solvers. We focus on strings here, but our approach carries over to other theories and their combinations. © 2020 Association for Computing Machinery

Identifying Overly Restrictive Matching Patterns in SMT-Based Program Verifiers

ISSN:0302-9743ISSN:1611-334