Enablers and Impediments for Collaborative Research in Software Testing: An Empirical Exploration

When it comes to industrial organizations, current collaboration efforts in software engineering research are very often kept in-house, depriving these organizations off the skills necessary to build independent collaborative research. The current trend, towards empirical software engineering research, requires certain standards to be established which would guide these collaborative efforts in creating a strong partnership that promotes independent, evidence-based, software engineering research. This paper examines key enabling factors for an efficient and effective industry-academia collaboration in the software testing domain. A major finding of the research was that while technology is a strong enabler to better collaboration, it must be complemented with industrial openness to disclose research results and the use of a dedicated tooling platform. We use as an example an automated test generation approach that has been developed in the last two years collaboratively with Bombardier Transportation AB in Sweden

Combinatorial modeling and test case generation for industrial control software using ACTS

Combinatorial testing has been suggested as an effective method of creating test cases at a lower cost. However, industrially applicable tools for modeling and combinatorial test generation are still scarce. As a direct effect, combinatorial testing has only seen a limited uptake in industry that calls into question its practical usefulness. This lack of evidence is especially troublesome if we consider the use of combinatorial test generation for industrial safety-critical control software, such as are found in trains, airplanes, and power plants. To study the industrial application of combinatorial testing, we evaluated ACTS, a popular tool for combinatorial modeling and test generation, in terms of applicability and test efficiency on industrial-sized IEC 61131-3 industrial control software running on Programmable Logic Controllers (PLC). We assessed ACTS in terms of its direct applicability in combinatorial modeling of IEC 61131-3 industrial software and the efficiency of ACTS in terms of generation time and test suite size. We used 17 industrial control programs provided by Bombardier Transportation Sweden AB and used in a train control management system. Our results show that not all combinations of algorithms and interaction strengths could generate a test suite within a realistic cut-off time. The results of the modeling process and the efficiency evaluation of ACTS are useful for practitioners considering to use combinatorial testing for industrial control software as well as for researchers trying to improve the use of such combinatorial testing techniques

Extension: Invited for a special contribution to the

Abstract During the last decade, testing with model-checking techniques for software-intensive systems has been developed based on the theory of model-checking. However, the main problem in using model-checking for testing industrial software systems is the potential combinatorial explosion of the state space and its limited application to models used in practice. In this thesis, we improve the current status of testing with model-checking techniques by developing a framework suitable for transforming Function Block Diagrams (FBD), a widely used model in safety-critical software applications, to a formal representation of both its functional and timing behavior. For this, we implement an automatic model-to-model transformation to timed automata. The transformation accurately reflects the data-flow characteristics of the FBD language by constructing a complete behavioral model which assumes a read-execute-write program semantics. In addition, we develop a test case generation technique based on model-checking, tailored for structural coverage of FBD programs. We define logic coverage for FBD programs based on the transformed timed automata model. This copes with both functional and timing behavior of an FBD program. This formal definition is necessary for the approach to be applicable to model-checking. We present how a model-checker can be used to generate test cases for covering an FBD program. The developed techniques have been implemented in a testing tool. To demonstrate the potential applications of our techniques, we present a framework for testing FBD programs and a case study where the tool and its methodology are applied. Based on our experiments, this method is -for the real world models provided by Bombardier Transportation AB -a useful and applicable way of generating test cases. Chapter 1 Background and Motivation Within the last decade model-checking has turned out to be a useful technique for generation of test cases from finite-state models Safety-critical and real-time software systems implemented in Programmable Logic Controllers (PLCs) are used in many real-world industrial application domains. One of the programming languages defined by the International Electrotechnical Commission (IEC) for PLCs is the Function Block Diagram (FBD). Programs developed in FBD are transformed into program code, which is compiled into machine code automatically by using specific engineering tools provided by PLC vendors. The motivation for using FBD as an implementation model comes from the fact that this language is the standard in many industrial software systems, such as rail transport control. In this thesis, our goal is to help testers automatically develop tests for safety-critical software systems modeled in FBD. One example includes logic coverage which needs to be demonstrated on the developed programs. There has been little research on using logic coverage criteria for FBD programs in an industrial setting. In some cases logic coverage is analyzed at the code level As the first contribution of this thesis, we developed a framework suitable for transforming FBD programs to a formal representation of both its functional and timing behavior. For this, we implement an automatic model-to-model transformation to timed automata, a well known model introduced by Alur and Dill As the second contribution of this thesis, we developed a testing technique based on modelchecking, tailored for logic coverage of FBD programs. There have been a number of testing techniques used for defining logic coverage using model-checkers, e.g., As the third contribution of this thesis, we developed a testing tool for safety critical applications, described in Function Block Diagram (FBD) language, aimed to support both a model and a searchbased approach. We found that FBD programs have many easy to cover structures that allow testing with model checkers to perform surprisingly well. Currently, we are investigating a more elaborate empirical evaluation for the use of testing with model checkers and logic coverage. Chapter 2 Research Description The past years have witnessed increasing research within model-based testing. The design of software has a direct impact on the final implementation, with respect to performance and other quality attributes. We argue that there is a need for testing safety-critical software that are originally described in the domain-specific language Function Block Diagram. We propose a model-based approach that integrates model checking, and describe its tool support. Thesis Statement In this section, we present the main goal of the thesis, which is split into three subgoals directly addressed in our work. Overall Goal. To provide an approach to software testing for Function Block Diagram models that is useful and applicable in practice. As we have described in Chapter 1, the need for testing Function Block Diagrams motivated us to provide a framework for testing of such models using model checking techniques. Thus, we chose it as our overall goal. Since this goal is too abstract to be directly addressed, we have further divide it into three more concrete subgoals. In order to be able to provide a framework for testing Function Block Diagrams, one needs an expressive and well-defined technique that would support testing functional and timing behavior. Hence we have formulated the first subgoal as follows: Subgoal 1. To present a framework for testing tailored to Function Block Diagrams. The first subgoal is the basis for the next two subgoals, in that it provides a model-based test generation method tailored for Function Block Diagram programs. The next step is to propose and demonstrate the use of the Uppaal tool for testing, which gives rise to the second subgoal as follows: Subgoal 2. To study and apply software testing on industrial systems using an integrated tool. In the second subgoal, we develop a testing tool based on the Uppaal model checker. Many benefits emerge from developing an integrated tool, including the ability to automatically generate test cases for real industrial software systems described in Function Block Diagram language. To support testers and developers when testing Function Block Diagram programs we have formulated the third subgoal as follows: Subgoal 3. To investigate how logic coverage can improve testing of Function Block Diagrams and which criteria are suitable for specific characteristics of real models. The last subgoal is based on the proposed logic coverage as an useful and applicable criteria to Function Block Diagram models and aims at providing evidence on the effectiveness and efficiency of logic coverage. Thesis Contribution The thesis will include four conference papers. In the first paper, we introduce the framework and by that we address Subgoal 1 and Subgoal 2. In Paper A we propose a translation of FBD programs into timed automata models. We present a test generation approach sing the UPPAAL model-checker in the context of a model-based approach towards unit testing. For the translation of an FBD program into a TA model, a set of rules are presented. On the basis of this model, a model checker has been used for generating consistent test suites. Paper B Based on Paper A and aimed at increasing confidence on the results for Subgoal 1 the second paper presents a testing tool for Function Block Diagrams, as well as several specific implications. The tool is aimed at safety critical applications described in Function Block Diagram language, and supports both a model and a search-based approach. In Paper B, and to achieve Subgoal 2, we describe the architecture of the tool, its workflow process, and a case study in which the tool has been applied in a real industrial setting to test a train control management system. Paper C As a direct result of the results from Paper A, we address Subgoal 3 in order to improve testing of Function Block Diagrams. We generate tests that cover the structure of Function Block Diagrams. One way of dealing with test generation is to approach it as a model checking problem, such that model checking tools automatically create tests. We start from the framework introduced in Paper A and we show how logic coverage criteria can be formalised and used by a model checker to provide test cases ready to be executed. From our experiments with a typical program we noticed that for more complicated logic coverage criteria, test cases result in longer traces than for simpler logic coverage criteria. To achieve Subgoal 2 we assess the applicability and scalability of using logic coverage for testing FBD programs with various sizes and specific complexities. For the models used in Paper C, the timer component appears to be significantly affecting the generation time. We modified the program by increasing or decreasing the number of components in the model. We note that the use of timer elements is influencing the handling of larger systems, with an increased cost of generation time and used memory. Paper D We conclude this collection of papers with a paper detailing a large case study, as well as a more elaborate empirical evaluation of the use of our framework. To further address Subgoal 3 we measure both efficiency and effectiveness of using logic coverage for Function Block Diagram programs. In Paper D, we empirically evaluate the fault detection effectiveness of logic coverage criteria using mutation analysis. We produce tests satisfying logic coverage criteria and generate mutants automatically for industrial Function Block Diagram models. Based on the results, we compare different logic criteria, and suggest improvements to testing Function Block Diagram. Research Methodology The research is based on both theoretical (Papers A-C) and empirical methodologies (Paper D) including deductive methods and analysis of quantitative data. In Papers A-C deductive research is performed by giving formal descriptions, using prototype implementations, and evaluating the framework on industrial examples. Publications Included in the Thesis This licentiate thesis is presented as a collection of papers. The following papers will be included in the thesis. Paper A Model-based Test Suite Generation for Function Block Diagrams using the UPPAAL Model Checker. Abstract. A method for model-based test generation of safety-critical applications using Programmable Logic Controllers and implemented in a programming language such as Function Block Diagram (FBD) is described. It involves the transformation of FBD programs with timed annotations into timed automata models which are used to automatically generate test suites. Specifically we demonstrate how to use model transformation for formalization and model-checking of FBD programs using the UPPAAL tool. Many benefits emerge from this method, including the ability to automatically generate test suites from a formal model in order to ensure compliance to strict quality requirements including unit testing and specific coverage measurements. The approach is experimentally assessed on a train control system in terms of consumed resources. 55 -60, ISBN: 978-1-4673-6284-9, 2013, IEEE. Abstract. In this paper we present a new testing tool for safety critical applications described in Function Block Diagram (FBD) language aimed to support both a model and a search-based approach. Many benefits emerge from this tool, including the ability to automatically generate test suites from an FBD program in order to comply to quality requirements such as component testing and specific coverage measurements. Search-based testing methods are used to generate test data based on executable code rather than the FBD program, alleviating any problems that may arise from the ambiguities that occur while creating FBD programs. Test cases generated by both approaches are executed and used as a way of cross validation. In the current work, we describe the architecture of the tool, its workflow process, and a case study in which the tool has been applied in a real industrial setting to test a train control management system. I am the main author and driver of this paper. I implemented the model-based testing part of the tool and performed the experiments. Abstract. In model-driven development, testers are often focusing on functional model-level testing, enabling verification of design models against their specifications. In addition, in safety-critical software development, testers are required to show that tests cover the structure of the implementation. Testing cost and time savings could be achieved if the process of deriving test cases for logic coverage is automated and provided test cases are ready to be executed. The logic coverage artifacts, i.e., predicates and clauses, are required for different logic coverage, e.g., MC/DC. One way of dealing with test case generation for ensuring logic coverage is to approach it as a model-checking problem, such that model-checking tools automatically create test cases. We show how logic coverage criteria can be formalized and used by a model-checker to provide test cases for ensuring this coverage on safety-critical software described in the Function Block Diagram programming language. Based on our experiments, this approach, supported by a tool chain, is an applicable and useful way of generating test cases for covering Function Block Diagrams. I am the main author of the paper, with my co-authors having academic and industrial advisory role. I implemented the models, the model transformation, and performed the experiments. Elaine Weyuker and Tom Ostrand took part in the discussions and contributed with improving parts of the paper. Abstract. Function Block Diagram, one of the PLC programming languages, is a widely used language to implement safety-critical software. We previously proposed logic coverage as useful and applicable to Function Block Diagram models. However important questions remain: How effective is logic coverage in terms of fault detection? In this paper, we empirically evaluate the fault detection effectiveness of logic coverage criteria using mutation analysis. We produce tests satisfying logic coverage criteria and generate mutants automatically for industrial Function Block Diagram models. Based on the results, we compare different logic criteria, and suggest improvements to testing Function Block Diagram

Extending EAST-ADL for modeling and analysis of system's resource-usage

EAST-ADL is an architectural description language dedicated to automotive embedded systems design, with focus on structural and functional modeling. The current architectural notations lack support for modeling and analysis of resource-usage, and therefore it is not possible to reason about resource requirements. In this paper, we describe our work towards filling the gap between EAST-ADL language and formal modeling and analysis of system’s resource usage, by extending the EAST-ADL language with embedded resources, such as storage, energy, communication and computation. To formalize this approach and provide a basis for rigorous analysis, we show how to analyze EAST-ADL models using the framework of priced timed automata and weighted CTL. We report our experiences from applying this approach for integrating resource-wise analysis into EAST-ADL

Using Timed Base-Choice Coverage Criterion for Testing Industrial Control Software

The base-choice criterion was proposed as a suitable technique for testing software based on its nominal choice of input parameters. Test cases are created based on this strategy by varying the values of one input parameter at a time while keeping the values of the other parameters fixed on the base choice. However, this strategy might not be as effective when used on industrial control software for testing timed behavior. We propose to incorporate time as another parameter when generating and executing tests by defining the timed base-choice coverage criterion. We performed an empirical evaluation using 11 industrial programs written in the IEC 61131-3 programming language. We found that tests generated for timed base-choice criterion show better code coverage (7% improvement) and fault detection (27% improvement) in terms of mutation score than tests satisfying base-choice coverage criterion. The results demonstrate the feasibility of applying timed base-choice criterion for testing industrial control software

Model-based Test Suite Generation for Function Block Diagrams using the UPPAAL Model Checker

A method for model-based test generation of safety-critical embedded applications using Programmable Logic Controllers and implemented in a programming language such as Function Block Diagram (FBD) is described. The FBD component model is based on the IEC 1131 standard and it is used primarily for embedded systems, in which timeliness is an important property to be tested. Our method involves the transformation of FBD programs with timed annotations into timed automata models which are used to automatically generate test suites. Specifically we demonstrate how to use model transformation for formalization and model checking of FBD programs using the UPPAAL tool. Many benefits emerge from this method, including the ability to automatically generate test suites from a formal model in order to ensure compliance to strict quality requirements including unit testing and specific coverage measurements. The approach is experimentally assessed on a train control system in terms of consumed resources

Using Logic Coverage to Improve Testing Function Block Diagrams

In model-driven development, testers are often focusing on functional model-level testing, enabling verification of design models against their specifications. In addition, in safety-critical software development, testers are required to show that tests cover the structure of the implementation. Testing cost and time savings could be achieved if the process of deriving test cases for logic coverage is automated and provided test cases are ready to be executed. The logic coverage artifacts, i.e., predicates and clauses, are required for different logic coverage, e.g., MC/DC. One way of dealing with test case generation for ensuring logic coverage is to approach it as a model-checking problem, such that model-checking tools automatically create test cases. We show how logic coverage criteria can be formalized and used by a model-checker to provide test cases for ensuring this coverage on safety-critical software described in the Function Block Diagram programming language. Based on our experiments, this approach, supported by a tool chain, is an applicable and useful way of generating test cases for covering Function Block Diagrams.25th IFIP WG 6.1 International Conference, ICTSS 2013, Istanbul, Turkey, November 13-15, 2013, ProceedingsATAC - Advanced Test Automation for Complex Software-Intensive System (ITEA2/Vinnova

Statistical Analysis of Resource Usage of Embedded Systems Modeled in EAST-ADL

The growing complexity of modern automotive embedded systems requires new techniques for model-based design that takes under consideration both software and hardware constraints and enables verification at early stages of development. In this context, EAST-ADL was developed as a domain specific language dedicated to modeling of functional-, software-, and hardware- architecture of automotive embedded systems. This language represents a convenient abstraction when reasoning about the system functionality and supports modeling of relevant extra-functional properties, like timing and resource usage. By providing formal semantics to the EAST-ADL language, as a network of priced timed automata, it becomes possible to reason about feasibility and worst-case resource consumption of the embedded components. In this paper, we show how to analyze such embedded systems modeled in EAST-ADL by using statistical model-checking. We report our experiences from applying this approach to an industrial Brake-by-Wire system.ITS-EASY Post Graduate School for Embedded Software and SystemsVeriSpec - Structured Specification and Automated Verification for Automotive Functional SafetyTOCSYC - Testing of Critical System Characteristics (KKS