Generating Effective Test Suites for Model Transformations Using Classifying Terms

Generating sample models for testing a model transformation is no easy task. This paper explores the use of classifying terms and stratified sampling for developing richer test cases for model transformations. Classifying terms are used to define the equivalence classes that characterize the relevant subgroups for the test cases. From each equivalence class of object models, several representative models are chosen depending on the required sample size. We compare our results with test suites developed using random sampling, and conclude that by using an ordered and stratified approach the coverage and effectiveness of the test suite can be significantly improved.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech

Spectrum-Based Fault Localization in Model Transformations

Model transformations play a cornerstone role in Model-Driven Engineering (MDE), as they provide the essential mechanisms for manipulating and transforming models. The correctness of software built using MDE techniques greatly relies on the correctness of model transformations. However, it is challenging and error prone to debug them, and the situation gets more critical as the size and complexity of model transformations grow, where manual debugging is no longer possible. Spectrum-Based Fault Localization (SBFL) uses the results of test cases and their corresponding code coverage information to estimate the likelihood of each program component (e.g., statements) of being faulty. In this article we present an approach to apply SBFL for locating the faulty rules in model transformations. We evaluate the feasibility and accuracy of the approach by comparing the effectiveness of 18 different stateof- the-art SBFL techniques at locating faults in model transformations. Evaluation results revealed that the best techniques, namely Kulcynski2, Mountford, Ochiai, and Zoltar, lead the debugger to inspect a maximum of three rules to locate the bug in around 74% of the cases. Furthermore, we compare our approach with a static approach for fault localization in model transformations, observing a clear superiority of the proposed SBFL-based method.Comisión Interministerial de Ciencia y Tecnología TIN2015-70560-RJunta de Andalucía P12-TIC-186

Bounded Exhaustive Search of Alloy Specification Repairs

The rising popularity of declarative languages and the hard to debug nature thereof have motivated the need for applicable, automated repair techniques for such languages. However, despite significant advances in program repair for imperative languages, there is a dearth of repair techniques for declarative languages. We present BeAFix, an automated repair technique for faulty models written in Alloy, a first-order relational logic language. BeAFix has a number of distinguishing features. Firstly, it supports any kind of oracle, including assertions typically found in formal specifications, as well as “specification tests”. This is important since unit tests, widely available for programs, are not typically found in formal specifications. Secondly, given a defined set of mutation operations, a set of suspicious expressions and a maximum number of mutations to apply per expression, BeAFix's bounded-exhaustive approach will either find a fix, or guarantee that such a fix is not possible, within the provided bounds. With respect to fault localization, BeAFix is not tightly integrated to any specific technique. In fact, our technique is independent of the fault localization technique used, and the fault localization is run only once before the repair process begins. To support a bounded-exhaustive approach while keeping repair times reasonable, sound state pruning techniques (i.e., those that guarantee that no valid fixes are removed) are introduced. When a faulty model has more than one suspicious expression, we use both syntactic analysis and dynamically generated scenario-based assertions to check the feasibility of a particular repair candidate. A failing check will determine that this candidate would never lead to a fully repaired model, allowing BeAFix to prune significant parts of the search space. We evaluated our technique on two Alloy benchmarks, including one previously used in a state-of-the-art technique for Alloy repair. The results show that BeAFix is able to repair thousands of real-world faulty models, corroborating its ability to effectively, and efficiently generate correct repairs while also being less prone to overfitting than previous techniques.Sociedad Argentina de Informática e Investigación Operativ

Applying Formal Methods to Networking: Theory, Techniques and Applications

Despite its great importance, modern network infrastructure is remarkable for the lack of rigor in its engineering. The Internet which began as a research experiment was never designed to handle the users and applications it hosts today. The lack of formalization of the Internet architecture meant limited abstractions and modularity, especially for the control and management planes, thus requiring for every new need a new protocol built from scratch. This led to an unwieldy ossified Internet architecture resistant to any attempts at formal verification, and an Internet culture where expediency and pragmatism are favored over formal correctness. Fortunately, recent work in the space of clean slate Internet design---especially, the software defined networking (SDN) paradigm---offers the Internet community another chance to develop the right kind of architecture and abstractions. This has also led to a great resurgence in interest of applying formal methods to specification, verification, and synthesis of networking protocols and applications. In this paper, we present a self-contained tutorial of the formidable amount of work that has been done in formal methods, and present a survey of its applications to networking.Comment: 30 pages, submitted to IEEE Communications Surveys and Tutorial

Automated synthesis and debugging of declarative models in alloy

In theory, formal specifications offer numerous benefits in developing more reliable software. In practice however, the use of specifications is rather limited, and practitioners often consider them more trouble than they are worth. Indeed, manually writing detailed specifications using notations that have unfamiliar syntax and semantics can be a daunting task -- even for experienced programmers. We introduce a new automated approach for synthesis of desired specifications and debugging of faulty specifications using given examples that capture the essence of desired properties and serve as test cases. Our focus is specifications written in the declarative language Alloy -- a first-order logic based on relations with transitive closure, and its SAT-based analysis engine. Our key insight is that a test-driven foundation enables modern approaches to synthesis and debugging of imperative code to serve as a basis for developing novel analogous techniques for declarative specifications. For synthesis, we build on equivalence in relational algebra and introduce techniques for generating candidate Alloy expressions. We also introduce a technique to complete a partial Alloy model with holes using constraint solving. For locating faults in buggy specifications, we build on mutation-based fault localization and introduce techniques for locating likely faulty nodes in the abstract syntax tree of the faulty specification. Moreover, we integrate our expression generation and fault localization techniques to introduce a technique for automated specification repair. We experimentally evaluate our techniques using several Alloy models as subjects, including those with real faults. The results show that our techniques are effective at synthesis and debugging of the subjects. We believe our techniques provide an important step towards increasing the role of formal specifications in developing more reliable software and realizing their promise.Electrical and Computer Engineerin

FLACK: Counterexample-Guided Fault Localization for Alloy Models

Alloy is a specification language that has been used in a wide range of applications, such as program verification, test case generation, IoT and Android security, etc. Unlike imperative languages, such as C or Java, Alloy is declarative, which describes the logic of a computation without describing its control flow and does not generate traces during the execution. Thus, traditional fault localization techniques developed for imperative programs based on analyzing the control flows of passing and failing tests do not directly apply to Alloy. To aid developers in debugging Alloy models, we develop FLACK, a tool to automatically localize Alloy buggy expressions. Given an Alloy model with violated assertions, FLACK automatically outputs a ranking list of expressions based on their spaciousness to the assertions violations. For each assertion, FLACK first queries the Alloy analyzer for counterexamples, i.e. instances of the model that violate the asserted property. FLACK then uses a Partial Max-SAT (PMAXSAT) solver to find instances that satisfy the asserted property and are most similar to the counterexamples. FLACK then identifies the relations and atoms that are different between the counterexamples and the satisfying instances. The differences illustrate how the counterexamples violate the assertion. The PMAXSAT solver guarantees that these differences are “minimal”, containing only essential information related to the assertion violation. By finding expressions most related to these differences, FLACK identifies the potential expressions causing the assertion violation. FLACK is different than the state of the art on Alloy fault localization in that it does not rely on unit tests which are not commonly found accompanying Alloy models. Instead, FLACK relies on assertions and constraint solvers to obtain counterexamples and satisfying instances, which are the main underlying technology in Alloy and commonly used by the Alloy developers.Sociedad Argentina de Informática e Investigación Operativ

Crucible: Graphical Test Cases for Alloy Models

Alloy is a declarative modeling language that is well suited for verifying system designs. Alloy models are automatically analyzed using the Analyzer, a toolset that helps the user understand their system by displaying the consequences of their properties, helping identify any missing or incorrect properties, and exploring the impact of modifications to those properties. To achieve this, the Analyzer invokes off-the-shelf SAT solvers to search for scenarios, which are assignments to the sets and relations of the model such that all executed formulas hold. To help write more accurate software models, Alloy has a unit testing framework, AUnit, which allows users to outline specific scenarios and check if those scenarios are correctly generated or prevented by their model. Unfortunately, AUnit currently only supports textual specifications of scenarios. This paper introduces Crucible, which allows users to graphically create AUnit test cases. In addition, Crucible provides automated guidance to users to ensure they are creating well structured, valuable test cases. As a result, Crucible eases the burden of adopting AUnit and brings AUnit test case creation more in line with how Alloy scenarios are commonly interacted with, which is graphically

Automatic Software Repair: a Bibliography

This article presents a survey on automatic software repair. Automatic software repair consists of automatically finding a solution to software bugs without human intervention. This article considers all kinds of repairs. First, it discusses behavioral repair where test suites, contracts, models, and crashing inputs are taken as oracle. Second, it discusses state repair, also known as runtime repair or runtime recovery, with techniques such as checkpoint and restart, reconfiguration, and invariant restoration. The uniqueness of this article is that it spans the research communities that contribute to this body of knowledge: software engineering, dependability, operating systems, programming languages, and security. It provides a novel and structured overview of the diversity of bug oracles and repair operators used in the literature

Systematic techniques for more effective fault localization and program repair

Debugging faulty code is a tedious process that is often quite expensive and can require much manual effort. Developers typically perform debugging in two key steps: (1) fault localization, i.e., identifying the location of faulty line(s) of code; and (2) program repair, i.e., modifying the code to remove the fault(s). Automating debugging to reduce its cost has been the focus of a number of research projects during the last decade, which have introduced a variety of techniques. However, existing techniques suffer from two basic limitations. One, they lack accuracy to handle real programs. Two, they focus on automating only one of the two key steps, thereby leaving the other key step to the developer. Our thesis is that an approach that integrates systematic search based on state-of-the-art constraint solvers with techniques to analyze artifacts that describe application specific properties and behaviors, provides the basis for developing more effective debugging techniques. We focus on faults in programs that operate on structurally complex inputs, such as heap-allocated data or relational databases. Our approach lays the foundation for a unified framework for localization and repair of faults in programs. We embody our thesis in a suite of integrated techniques based on propositional satisfiability solving, correctness specifications analysis, test-spectra analysis, and rule-learning algorithms from machine learning, implement them as a prototype tool-set, and evaluate them using several subject programs.Electrical and Computer Engineerin