Constraint-Based Heuristic On-line Test Generation from Non-deterministic I/O EFSMs

We are investigating on-line model-based test generation from non-deterministic output-observable Input/Output Extended Finite State Machine (I/O EFSM) models of Systems Under Test (SUTs). We propose a novel constraint-based heuristic approach (Heuristic Reactive Planning Tester (xRPT)) for on-line conformance testing non-deterministic SUTs. An indicative feature of xRPT is the capability of making reasonable decisions for achieving the test goals in the on-line testing process by using the results of off-line bounded static reachability analysis based on the SUT model and test goal specification. We present xRPT in detail and make performance comparison with other existing search strategies and approaches on examples with varying complexity.Comment: In Proceedings MBT 2012, arXiv:1202.582

Using formal methods to support testing

Formal methods and testing are two important approaches that assist in the development of high quality software. While traditionally these approaches have been seen as rivals, in recent years a new consensus has developed in which they are seen as complementary. This article reviews the state of the art regarding ways in which the presence of a formal specification can be used to assist testing

Extended Finite State Machine based test generation for an OpenFlow switch

Implementations of an OpenFlow (OF) switch, a crucial Software Defined Networking (SDN) component, are prone to errors caused by developer mistakes or/and ambiguous requirements stated in the OF documents. The paper is devoted to test derivation for related OF switch implementations. A model based test generation strategy is proposed. It relies on an Extended Finite State Machine (EFSM) specification that describes the functional behaviour of the switch-to-controller communication while potential faults/misconfigurations are expressed via corresponding mutation operators. We propose a method for deriving a test suite that contains distinguishing sequences for the specification EFSM and corresponding mutants. The proposed approach is implemented as a testbed to automatically derive and execute the test suites against different versions of an OF implementation. Preliminary experimental evaluation has shown the effectiveness of the proposed approach. Further on, the derived test suites have been able to detect a number of functional inconsistencies such as erroneous responses to the Flow Mod adding rules with specific 'action' values in an available Open vSwitch implementatio

Improving Software Quality by Synergizing Effective Code Inspection and Regression Testing

Software quality assurance is an essential practice in software development and maintenance. Evolving software systems consistently and safely is challenging. All changes to a system must be comprehensively tested and inspected to gain confidence that the modified system behaves as intended. To detect software defects, developers often conduct quality assurance activities, such as regression testing and code review, after implementing or changing required functionalities. They commonly evaluate a program based on two complementary techniques: dynamic program analysis and static program analysis. Using an automated testing framework, developers typically discover program faults by observing program execution with test cases that encode required program behavior as well as represent defects. Unlike dynamic analysis, developers make sure of the program correctness without executing a program by static analysis. They understand source code through manual inspection or identify potential program faults with an automated tool for statically analyzing a program. By removing the boundaries between static and dynamic analysis, complementary strengths and weaknesses of both techniques can create unified analyses. For example, dynamic analysis is efficient and precise but it requires selection of test cases without guarantee that the test cases cover all possible program executions, and static analysis is conservative and sound but it produces less precise results due to its approximation of all possible behaviors that may perform at run time. Many dynamic and static techniques have been proposed, but testing a program involves substantial cost and risks and inspecting code change is tedious and error-prone. Our research addresses two fundamental problems in dynamic and static techniques. (1) To evaluate a program, developers are typically required to implement test cases and reuse them. As they develop more test cases for verifying new implementations, the execution cost of test cases increases accordingly. After every modification, they periodically conduct regression test to see whether the program executes without introducing new faults in the presence of program evolution. To reduce the time required to perform regression testing, developers should select an appropriate subset of the test suite with a guarantee of revealing faults as running entire test cases. Such regression testing selection techniques are still challenging as these methods also have substantial costs and risks and discard test cases that could detect faults. (2) As a less formal and more lightweight method than running a test suite, developers often conduct code reviews based on tool support; however, understanding context and changes is the key challenge of code reviews. While reviewing code changes—addressing one single issue—might not be difficult, it is extremely difficult to understand complex changes—including multiple issues such as bug fixes, refactorings, and new feature additions. Developers need to understand intermingled changes addressing multiple development issues, finding which region of the code changes deals with a particular issue. Although such changes do not cause trouble in implementation, investigating these changes becomes time-consuming and error-prone since the intertwined changes are loosely related, leading to difficulty in code reviews. To address the limitations outlined above, our research makes the following contributions. First, we present a model-based approach to efficiently build a regression test suite that facilitates Extended Finite State Machines (EFSMs). Changes to the system are performed at transition level by adding, deleting or replacing transition. Tests are a sequence of input and expected output messages with concrete parameter values over the supported data types. Fully-observable tests are introduced whose descriptions contain all the information about the transitions executed by the tests. An invariant characterizing fully observable tests is formulated such that a test is fully-observable whenever the invariant is a satisfiable formula. Incremental procedures are developed to efficiently evaluate the invariant and to select tests from a test suite that are guaranteed to exercise a given change when the tests run on a modified EFSM. Tests rendered unusable due to a change are also identified. Overlaps among the test descriptions are exploited to extend the approach to simultaneously select and discard multiple tests to alleviate the test selection costs. Although test regression selection problem is NP-hard [78], the experimental results show the cost of our test selection procedure is still acceptable and economical. Second, to support code review and regression testing, we present a technique, called ChgCutter. It helps developers understand and validate composite changes as follows. It interactively decomposes these complex, composite changes into atomic changes, builds related change subsets using program dependence relationships without syntactic violation, and safely selects only related test cases from the test suite to reduce the time to conduct regression testing. When a code reviewer selects a change region from both original and changed versions of a program, ChgCutter automatically identifies similar change regions based on the dependence analysis and the tree-based code search technique. By automatically applying a change to the identified regions in an original program version, ChgCutter generates a program version which is a syntactically correct version of program. Given a generated program version, it leverages a testing selection technique to select and run a subset of the test suite affected by a change automatically separated from mixed changes. Based on the iterative change selection process, there can be each different program version that include its separated change. Therefore, ChgCutter helps code reviewers inspect large, complex changes by effectively focusing on decomposed change subsets. In addition to assisting understanding a substantial change, the regression testing selection technique effectively discovers defects by validating each program version that contains a separated change subset. In the evaluation, ChgCutter analyzes 28 composite changes in four open source projects. It identifies related change subsets with 95.7% accuracy, and it selects test cases affected by these changes with 89.0% accuracy. Our results show that ChgCutter should help developers effectively inspect changes and validate modified applications during development

Realizability of embedded controllers: from hybrid models to correct implementations

Un controller embedded \ue8 un dispositivo (ovvero, un'opportuna combinazione di componenti hardware e software) che, immerso in un ambiente dinamico, deve reagire alle variazioni ambientali in tempo reale. I controller embedded sono largamente adottati in molti contesti della vita moderna, dall'automotive all'avionica, dall'elettronica di consumo alle attrezzature mediche. La correttezza di tali controller \ue8 indubbiamente cruciale. Per la progettazione e per la verifica di un controller embedded, spesso sorge la necessit\ue0 di modellare un intero sistema che includa sia il controller, sia il suo ambiente circostante. La natura di tale sistema \ue8 ibrido. Esso, infatti, \ue8 ottenuto integrando processi ad eventi discreti (i.e., il controller) e processi a tempo continuo (i.e., l'ambiente). Sistemi di questo tipo sono chiamati cyber-physical (CPS) o sistemi ibridi. Le dinamiche di tali sistemi non possono essere rappresentati efficacemente utilizzando o solo un modello (i.e., rappresentazione) discreto o solo un modello continuo. Diversi tipi di modelli possono sono stati proposti per descrivere i sistemi ibridi. Questi si concentrano su obiettivi diversi: modelli dettagliati sono eccellenti per la simulazione del sistema, ma non sono adatti per la sua verifica; modelli meno dettagliati sono eccellenti per la verifica, ma non sono convenienti per i successivi passi di raffinamento richiesti per la progettazione del sistema, e cos\uec via. Tra tutti questi modelli, gli Automi Ibridi (HA) [8, 77] rappresentano il formalismo pi\uf9 efficace per la simulazione e la verifica di sistemi ibridi. In particolare, un automa ibrido rappresenta i processi ad eventi discreti per mezzo di macchine a stati finiti (FSM), mentre i processi a tempo continuo sono rappresentati mediante variabili "continue" la cui dinamica \ue8 specificata da equazioni differenziali ordinarie (ODE) o loro generalizzazioni (e.g., inclusioni differenziali). Sfortunatamente, a causa della loro particolare semantica, esistono diverse difficolt\ue0 nel raffinare un modello basato su automi ibridi in un modello realizzabile e, di conseguenza, esistono difficolt\ue0 nell'automatizzare il flusso di progettazione di sistemi ibridi a partire da automi ibridi. Gli automi ibridi, infatti, sono considerati dispositivi "perfetti e istantanei". Essi adottano una nozione di tempo e di variabili basata su insiemi "densi" (i.e., l'insieme dei numeri reali). Pertanto, gli automi ibridi possono valutare lo stato (i.e., i valori delle variabili) del sistema in ogni istante, ovvero in ogni infinitesimo di tempo, e con la massima precisione. Inoltre, sono in grado di eseguire computazioni o reagire ad eventi di sincronizzazione in modo istantaneo, andando a cambiare la modalit\ue0 di funzionamento del sistema senza alcun ritardo. Questi aspetti sono convenienti a livello di modellazione, ma nessun dispositivo hardware/software potrebbe implementare correttamente tali comportamenti, indipendentemente dalle sue prestazioni. In altre parole, il controller modellato potrebbe non essere implementabile, ovvero, esso potrebbe non essere realizzabile affatto. Questa tesi affronta questo problema proponendo una metodologia completa e gli strumenti necessari per derivare da modelli basati su automi ibridi, modelli realizzabili e le corrispondenti implementazioni corrette. In un modello realizzabile, il controller analizza lo stato del sistema ad istanti temporali discreti, tipicamente fissati dalla frequenza di clock del processore installato sul dispositivo che implementa il controller. Lo stato del sistema \ue8 dato dai valori delle variabili rilevati dai sensori. Questi valori vengono digitalizzati con precisione finita e propagati al controller che li elabora per decidere se cambiare la modalit\ue0 di funzionamento del sistema. In tal caso, il controller genera segnali che, una volta trasmessi agli attuatori, determineranno il cambiamento della modalit\ue0 di funzionamento del sistema. \uc8 necessario tener presente che i sensori e gli attuatori introducono ritardi che seppur limitati, non possono essere trascurati.An embedded controller is a reactive device (e.g., a suitable combination of hardware and software components) that is embedded in a dynamical environment and has to react to environment changes in real time. Embedded controllers are widely adopted in many contexts of modern life, from automotive to avionics, from consumer electronics to medical equipment. Noticeably, the correctness of such controllers is crucial. When designing and verifying an embedded controller, often the need arises to model the controller and also its surrounding environment. The nature of the obtained system is hybrid because of the inclusion of both discrete-event (i.e., controller) and continuous-time (i.e., environment) processes whose dynamics cannot be characterized faithfully using either a discrete or continuous model only. Systems of this kind are named cyber-physical (CPS) or hybrid systems. Different types of models may be used to describe hybrid systems and they focus on different objectives: detailed models are excellent for simulation but not suitable for verification, high-level models are excellent for verification but not convenient for refinement, and so forth. Among all these models, hybrid automata (HA) [8, 77] have been proposed as a powerful formalism for the design, simulation and verification of hybrid systems. In particular, a hybrid automaton represents discrete-event processes by means of finite state machines (FSM), whereas continuous-time processes are represented by using real-numbered variables whose dynamics is specified by (ordinary) differential equation (ODE) or their generalizations (e.g., differential inclusions). Unfortunately, when the high-level model of the hybrid system is a hybrid automaton, several difficulties should be solved in order to automate the refinement phase in the design flow, because of the classical semantics of hybrid automata. In fact, hybrid automata can be considered perfect and instantaneous devices. They adopt a notion of time and evaluation of continuous variables based on dense sets of values (usually R, i.e., Reals). Thus, they can sample the state (i.e., value assignments on variables) of the hybrid system at any instant in such a dense set R 650. Further, they are capable of instantaneously evaluating guard constraints or reacting to incoming events by performing changes in the operating mode of the hybrid system without any delay. While these aspects are convenient at the modeling level, any model of an embedded controller that relies for its correctness on such precision and instantaneity cannot be implemented by any hardware/software device, no matter how fast it is. In other words, the controller is un-realizable, i.e., un-implementable. This thesis proposes a complete methodology and a framework that allows to derive from hybrid automata proved correct in the hybrid domain, correct realizable models of embedded controllers and the related discrete implementations. In a realizable model, the controller samples the state of the environment at periodic discrete time instants which, typically, are fixed by the clock frequency of the processor implementing the controller. The state of the environment consists of the current values of the relevant variables as observed by the sensors. These values are digitized with finite precision and reported to the controller that may decide to switch the operating mode of the environment. In such a case, the controller generates suitable output signals that, once transmitted to the actuators, will effect the desired change in the operating mode. It is worth noting that the sensors will report the current values of the variables and the actuators will effect changes in the rates of evolution of the variables with bounded delays

Proceedings of Monterey Workshop 2001 Engineering Automation for Sofware Intensive System Integration

The 2001 Monterey Workshop on Engineering Automation for Software Intensive System Integration was sponsored by the Office of Naval Research, Air Force Office of Scientific Research, Army Research Office and the Defense Advance Research Projects Agency. It is our pleasure to thank the workshop advisory and sponsors for their vision of a principled engineering solution for software and for their many-year tireless effort in supporting a series of workshops to bring everyone together.This workshop is the 8 in a series of International workshops. The workshop was held in Monterey Beach Hotel, Monterey, California during June 18-22, 2001. The general theme of the workshop has been to present and discuss research works that aims at increasing the practical impact of formal methods for software and systems engineering. The particular focus of this workshop was "Engineering Automation for Software Intensive System Integration". Previous workshops have been focused on issues including, "Real-time & Concurrent Systems", "Software Merging and Slicing", "Software Evolution", "Software Architecture", "Requirements Targeting Software" and "Modeling Software System Structures in a fastly moving scenario".Office of Naval ResearchAir Force Office of Scientific Research Army Research OfficeDefense Advanced Research Projects AgencyApproved for public release, distribution unlimite

Model checking: Correct Web page navigations with browser behavior.

While providing better performance, transparency and expressiveness, the main features of the web technologies such as web caching, session and cookies, dynamically generated web pages etc. may also affect the correct understanding of the web applications running on top of them. From the viewpoint of formal verification and specification-based testing, this suggests that the formal model of the web application we use for static analysis or test case generation should contain the abstract behavior of the underlying web application environment. Here we consider the automated generation of such a model in terms of extended finite state machines from a given abstract description of a web application by incorporating the abstract behavioral model of the web browsers in the presence of session/cookies and dynamically generated web pages. The derived model can serve as the formal basis for both model checking and specification-based testing on the web applications where we take into account the effect of the internal caching mechanism to the correct accessibility of the web pages, which can be quite sensitive to the security of the information they carry. In order to check the correctness of the derived model against required properties, we provide the automated translation of the model into Promela. By applying SPIN on Promela models, we present experimental results on the evaluation of the proposed modeling in terms of scalability.Dept. of Computer Science. Paper copy at Leddy Library: Theses & Major Papers - Basement, West Bldg. / Call Number: Thesis2004 .Z543. Source: Masters Abstracts International, Volume: 43-05, page: 1761. Adviser: Jessica Chen. Thesis (M.Sc.)--University of Windsor (Canada), 2004

Exploring formal verification methodology for FPGA-based digital systems.

Abstract Not Provide

Validation and Verification of Safety-Critical Systems in Avionics

This research addresses the issues of safety-critical systems verification and validation. Safety-critical systems such as avionics systems are complex embedded systems. They are composed of several hardware and software components whose integration requires verification and testing in compliance with the Radio Technical Commission for Aeronautics standards and their supplements (RTCA DO-178C). Avionics software requires certification before its deployment into an aircraft system, and testing is mandatory for certification. Until now, the avionics industry has relied on expensive manual testing. The industry is searching for better (quicker and less costly) solutions. This research investigates formal verification and automatic test case generation approaches to enhance the quality of avionics software systems, ensure their conformity to the standard, and to provide artifacts that support their certification. The contributions of this thesis are in model-based automatic test case generations approaches that satisfy MC/DC criterion, and bidirectional requirement traceability between low-level requirements (LLRs) and test cases. In the first contribution, we integrate model-based verification of properties and automatic test case generation in a single framework. The system is modeled as an extended finite state machine model (EFSM) that supports both the verification of properties and automatic test case generation. The EFSM models the control and dataflow aspects of the system. For verification, we model the system and some properties and ensure that properties are correctly propagated to the implementation via mandatory testing. For testing, we extended an existing test case generation approach with MC/DC criterion to satisfy RTCA DO-178C requirements. Both local test cases for each component and global test cases for their integration are generated. The second contribution is a model checking-based approach for automatic test case generation. In the third contribution, we developed an EFSM-based approach that uses constraints solving to handle test case feasibility and addresses bidirectional requirements traceability between LLRs and test cases. Traceability elements are determined at a low-level of granularity, and then identified, linked to their source artifact, created, stored, and retrieved for several purposes. Requirements’ traceability has been extensively studied but not at the proposed low-level of granularity