Test Generation and Evaluation from High-Level Properties for Common Criteria Evaluations - The TASCCC Testing Tool

International audienceIn this paper, we present a model-based testing tool resulting from a research project, named TASCCC. This tool is a complete tool chain dedicated to property-based testing in UML/OCL, that integrates various technologies inside a dedicated Eclipse plug-in. The test properties are expressed in a dedicated language based on property patterns. These properties are then used for two purposes. First, they can be employed to evaluate the relevance of a test suite according to specific coverage criteria. Second, it is possible to generate test scenarios that will illustrate or exercise the property. These test scenarios are then unfolded and animated on the Smartesting's CertifyIt model animator, that is used to filter out infeasible sequences. This tool has been used in industrial partnership, aiming at providing an assistance for Common Criteria evaluations, especially by providing test generation reports used to show the link between the test cases and the Common Criteria artefacts

Software Test Case Generation Tools and Techniques: A Review

Software Industry is evolving at a very fast pace since last two decades. Many software developments, testing and test case generation approaches have evolved in last two decades to deliver quality products and services. Testing plays a vital role to ensure the quality and reliability of software products. In this paper authors attempted to conduct a systematic study of testing tools and techniques. Six most popular e-resources called IEEE, Springer, Association for Computing Machinery (ACM), Elsevier, Wiley and Google Scholar to download 738 manuscripts out of which 125 were selected to conduct the study. Out of 125 manuscripts selected, a good number approx. 79% are from reputed journals and around 21% are from good conference of repute. Testing tools discussed in this paper have broadly been divided into five different categories: open source, academic and research, commercial, academic and open source, and commercial & open source. The paper also discusses several benchmarked datasets viz. Evosuite 10, SF100 Corpus, Defects4J repository, Neo4j, JSON, Mocha JS, and Node JS to name a few. Aim of this paper is to make the researchers aware of the various test case generation tools and techniques introduced in the last 11 years with their salient features

DASE: Document-Assisted Symbolic Execution for Improving Automated Test Generation

Software testing is crucial for uncovering software defects and ensuring software reliability. Symbolic execution has been utilized for automatic test generation to improve testing effectiveness. However, existing test generation techniques based on symbolic execution fail to take full advantage of programs’ rich amount of documentation specifying their input constraints, which can further enhance the effectiveness of test generation. In this paper we present a general approach, Document-Assisted Symbolic Execution (DASE), to improve automated test generation and bug detection. DASE leverages natural language processing techniques and heuristics to analyze programs’ readily available documentation and extract input constraints. The input constraints are then used as pruning criteria; inputs far from being valid are trimmed off. In this way, DASE guides symbolic execution to focus on those inputs that are semantically more important. We evaluated DASE on 88 programs from 5 mature real-world software suites: GNU Coreutils, GNU findutils, GNU grep, GNU Binutils, and elftoolchain. Compared to symbolic execution without input constraints, DASE increases line coverage, branch coverage, and call coverage by 5.27–22.10%, 5.83–21.25% and 2.81–21.43% respectively. In addition, DASE detected 13 previously unknown bugs, 6 of which have already been confirmed by the developers.1 yea

A Specification-based Test Case Generation Method for UML/OCL

Abstract Automated test data generation is an important method for the verification and validation of UML/OCL specifications. In this paper, we present an extension of DNF-based test case generation methods to class models and recursive query operations on them. A key feature of our approach is an implicit representation of object graphs avoiding a representation based on object-id’s; thus, our approach avoids the generation of isomorphic object graphs by using a concise and still human-readable symbolic representation

Object Constraint Language Based Test Case Optimisation

Software testing, a pivotal phase in the Software Develop- ment Life Cycle (SDLC), ensures the correctness and perfor- mance of the corresponding software system. Model-based testing (MBT) is a method to validate whether the software system or specification satisfies the pre-defined requirements through design models. However, with modern systems ex- panding in complexity, testing has become a labour-intensive and unpredictable process within the SDLC. Therefore, many test case optimisation (TCO) techniques have been proposed to make the testing process more manageable. But these ap- proaches predominantly focus on code-based strategies, leav- ing systems expressed in Object Constraint Language (OCL) underserved. OCL is a part of the Unified Modeling Lan- guage (UML) standard and is a type of declarative language used to describe system specification by pre- and post- con- ditions. Initially, OCL has been proposed as a constraint language to add more details to the UML model, but along- side the development of OCL itself, there are more and more systems whose specifications are expressed in OCL.This thesis aims to systematically investigate the feasibil- ity of applying TCO techniques to the OCL-defined systems, with an emphasis on test case prioritisation (TCP) and test case minimisation (TCM) processes. A systematic literature review for the directly related topic, UML-based test case generation, is conducted in this thesis. Also, we adapted a set of test case optimisation algorithms and compared the performance between these algorithms under the context of OCL. Moreover, we modified one metric for the TCP evalu- ation process, which made the metric more suitable for the MBT and mutation testing environment. Furthermore, we introduce a full set of mutation operators and corresponding classifications to the OCL standard library, offering practical guidance for optimisation processes.The proposed TCO processes are validated and evalu- ated through four real-world systems expressed in OCL with different complexities. The experiment results demonstrate that for the TCM process, the size of the minimised test suite is reduced from 33.33% to 81.8% without losing any fault de- tection ability. For the TCP process, leveraging the modified evaluation metric, the improvements are up to 50%, indicat- ing that the prioritised test suite can detect system defects earlier when compared to the original one. Evaluating based on the considerations of effectiveness, efficiency and stability, we suggest the NSGA-II for the TCM process and the genetic algorithm for the TCP process. When combining TCP and TCM processes, the TCM process consistently increases the efficiency of the TCP process by reducing the search space for the prioritisation process

Improving Software Dependability through Documentation Analysis

Software documentation contains critical information that describes a system’s functionality and requirements. Documentation exists in several forms, including code comments, test plans, manual pages, and user manuals. The lack of documentation in existing software systems is an issue that impacts software maintainability and programmer productivity. Since some code bases contain a large amount of documentation, we want to leverage these existing documentation to improve software dependability. Specifically, we utilize documentation to help detect software bugs and repair corrupted files, which can reduce the number of software error and failure to improve a system’s reliability (e.g., continuity of correct service). We also generate documentation (e.g., code comment) automatically to help developers understand the source code, which helps improve a system’s maintainability (e.g., ability to undergo repairs and modifications). In this thesis, we analyze software documentation and propose two branches of work, which focuses on three types of documentation including manual pages, code comments, and user manuals. The first branch of work focuses on documentation analysis because documentation contains valuable information that describes the behavior of the program. We automatically extract constraints from documentation and apply them on a dynamic analysis symbolic execution tool to find bugs in the target software, and we extract constraints manually from documentation and apply them on a structured-file parsing application to repair corrupted PDF files. The second branch of work focuses on automatic code comment generation to improve software documentation. For documentation analysis, we propose and implement DASE and DocRepair. DASE leverages automatically extracted constraints from documentation to improve a dynamic analysis symbolic execution tool. DASE guides symbolic execution to focus the testing on execution paths that execute a program’s core functionalities using constraints learned from the documentation. We evaluated DASE on 88 programs from five mature real-world software suites to detect software bugs. DASE detects 12 previously unknown bugs that symbolic execution would fail to detect when given no input constraints, 6 of which have been confirmed by the developers. In DocRepair we perform an empirical study to study and repair corrupted PDF files. We create the first dataset of 319 corrupted PDF files and conduct an empirical study on 119 real-world corrupted PDF files to study the common types of file corruption. Based on the result of the empirical study we propose a technique called DocRepair. DocRepair’s repair algorithm includes seven repair operators that utilizes manually extracted constraints from documentation to repair corrupted files. We evaluate DocRepair against three common PDF repair tools. Amongst the 1,827 collected corrupted files from over two corpora of PDF files, DocRepair can successfully repair 354 files compared to Mutool, PDFtk, and GhostScript which repair 508, 41 and 84 respectively. We also propose a technique to combine multiple repair tools called DocRepair+, which can successfully repair 751 files. In the case where there is a lack of documentation, DASE and DocRepair+ would not work. Therefore, we propose automated documentation generation to address the issue. We propose and implement CloCom+ to generate code comments by mining both existing software repositories in GitHub and a Question and Answer site, Stack Overflow. CloCom+ generated 442 unique comments for 16 Java projects. Although CloCom+ improves on previous work, SumSlice, on automatic comment generation, the quality (evaluated on completeness, conciseness, expressiveness, and usefulness) and yield (number of generated comments) are still rather low which makes the technique not ready for real-world usage. In the future, it may be possible to combine the two proposed branches of work (documentation analysis and documentation generation) to further improve software dependability. For example, we can extract constraints from the automatically generated documentation (e.g., code comments)

Scalable Model-based Robustness Testing: Novel Methodologies and Industrial Application

Embedded systems, as for example communication and control systems, are being increasingly used in our daily lives and hence require thorough and systematic testing before their actual use. Many of these systems interact with their environment and, therefore, their functionality is largely dependent on this environment whose behavior can be unpredictable. Robustness testing aims at testing the behavior of a system in the presence of faulty situations in its operating environment (e.g., sensors and actuators). In such situations, the system should gracefully degrade its performance instead of abruptly stopping execution. To systematically perform robustness testing, one option is to resort to Model-Based Robustness Testing (MBRT), which is a systematic, rigorous, and automated way of conducting robustness testing. However, to successfully apply MBRT in industrial contexts, new technologies need to be developed to scale to the complexity of real industrial systems. This thesis presents a solution for MBRT on industrial systems, including scalable robustness modeling and executable test case generation. One important contribution of this thesis is a scalable RobUstness Modeling Methodology (RUMM), which is achieved using Aspect-Oriented Modeling (AOM). It is a complete, automated, and practical methodology that covers all features of state machines and aspect concepts necessary for MBRT. Such methodology, relying on a standard (Unified Modeling Language or UML) and using the target notation as the basis to model the aspects themselves, is expected to make the practical adoption of robustness modeling easier in industrial contexts. The applicability of the methodology is demonstrated using an industrial case study. Results showed that the approach significantly reduced modeling effort (98% on average), improved separation of concerns, and eased model evolution. The approach is further empirically evaluated using two controlled experiments involving human subjects and results showed that the proposed methodology significantly improves the readability of models as compared to modeling using standard UML notations. Another important contribution of this thesis is an efficient approach for solving constraints (written in Objects Constraint Language (OCL)) on the operating environment of a system, which is mandatory for emulating faulty situation in the environment for the purpose of MBRT. A set of novel heuristics is devised for various OCL constructs, which are required for the application of search algorithms. The heuristics have been empirically evaluated on an industrial case study for robustness testing and the results showed to be very promising and significantly better than the existing works in the literature on OCL constraint solvers. A final contribution of the thesis is robustness test case generation from the models developed using RUMM. Test case generation also includes scripts generation for environment emulation, which is mandatory for automated robustness testing again using an industrial case study. In preliminary experiments, the execution of test cases found one critical, robustness fault in a deployed industrial system