Evaluation of String Constraint Solvers Using Dynamic Symbolic Execution

Symbolic execution is a path sensitive program analysis technique used for error detection and test case generation. Symbolic execution tools rely on constraint solvers to determine the feasibility of program paths and generate concrete inputs for feasible paths. Therefore, the effectiveness of such tools depends on their constraint solvers. Most modern constraint solvers for primitive data types, such as integers, are both efficient and accurate. However, the research on constraint solvers for complex data types, such as strings, is ongoing and less converged. For example, there are several types of string constraint solvers provided by researchers. However, a potential user of a string constraint solver likely has no comprehensive means to identify which solver would work best for a particular problem. In order to help the user with selecting a solver, in addition to the commonly used performance criterion, we introduce two criteria: modeling cost and accuracy. Using these selection criteria, we evaluate four string constraint solvers in the context of symbolic execution. Our results show that, depending on the needs of the user, one solver might be more appropriate than another, yet no solver exhibits the best overall results. Hence, we suggest that the preferred approach to solving constraints for complex types is to execute all solvers in parallel and enable communication between solvers

User-Defined Backtracking Criteria for Symbolic Execution

Symbolic execution is a path-sensitive program analysis technique that aids users with program verification. To avoid exploring infeasible paths, symbolic execution checks the prefix of a current path for feasibility by adding a branch constraint to the path prefix and passing the formula to an off-the-shelf SMT solver for an evaluation. If the solver returns SAT/UNSAT, then the prefix is marked as feasible/infeasible. However, the solver can also return an UNKNOWN result, which means it cannot evaluate the formula. In addition, an operation occurring before a constraint can cause over-approximation that propagates to the solver\u27s result. Moreover, symbolic execution might time out the solver if it takes too long to run. A symbolic execution tool might handle these uncertainties by backtracking or by continuing its exploration of the prefix. This paper examines the behavior of path constraints beyond uncertain backtracking points. String and integer constraints are collected from concrete program execution via dynamic symbolic execution. These constraints are used to analyze how over- approximation in a path prefix affects the completeness of its extensions. We also examine variations in time required to decide a path constraint. Our findings suggest that a custom backtracking criteria defined by the user does improve the completeness of symbolic execution