5,935 research outputs found
Model based test suite minimization using metaheuristics
Software testing is one of the most widely used methods for quality assurance and fault detection purposes. However, it is one of the most expensive, tedious and time consuming activities in software development life cycle. Code-based and specification-based testing has been going on for almost four decades. Model-based testing (MBT) is a relatively new approach to software testing where the software models as opposed to other artifacts (i.e. source code) are used as primary source of test cases. Models are simplified representation of a software system and are cheaper to execute than the original or deployed system. The main objective of the research presented in this thesis is the development of a framework for improving the efficiency and effectiveness of test suites generated from UML models. It focuses on three activities: transformation of Activity Diagram (AD) model into Colored Petri Net (CPN) model, generation and evaluation of AD based test suite and optimization of AD based test suite. Unified Modeling Language (UML) is a de facto standard for software system analysis and design. UML models can be categorized into structural and behavioral models. AD is a behavioral type of UML model and since major revision in UML version 2.x it has a new Petri Nets like semantics. It has wide application scope including embedded, workflow and web-service systems. For this reason this thesis concentrates on AD models. Informal semantics of UML generally and AD specially is a major challenge in the development of UML based verification and validation tools. One solution to this challenge is transforming a UML model into an executable formal model. In the thesis, a three step transformation methodology is proposed for resolving ambiguities in an AD model and then transforming it into a CPN representation which is a well known formal language with extensive tool support. Test case generation is one of the most critical and labor intensive activities in testing processes. The flow oriented semantic of AD suits modeling both sequential and concurrent systems. The thesis presented a novel technique to generate test cases from AD using a stochastic algorithm. In order to determine if the generated test suite is adequate, two test suite adequacy analysis techniques based on structural coverage and mutation have been proposed. In terms of structural coverage, two separate coverage criteria are also proposed to evaluate the adequacy of the test suite from both perspectives, sequential and concurrent. Mutation analysis is a fault-based technique to determine if the test suite is adequate for detecting particular types of faults. Four categories of mutation operators are defined to seed specific faults into the mutant model. Another focus of thesis is to improve the test suite efficiency without compromising its effectiveness. One way of achieving this is identifying and removing the redundant test cases. It has been shown that the test suite minimization by removing redundant test cases is a combinatorial optimization problem. An evolutionary computation based test suite minimization technique is developed to address the test suite minimization problem and its performance is empirically compared with other well known heuristic algorithms. Additionally, statistical analysis is performed to characterize the fitness landscape of test suite minimization problems. The proposed test suite minimization solution is extended to include multi-objective minimization. As the redundancy is contextual, different criteria and their combination can significantly change the solution test suite. Therefore, the last part of the thesis describes an investigation into multi-objective test suite minimization and optimization algorithms. The proposed framework is demonstrated and evaluated using prototype tools and case study models. Empirical results have shown that the techniques developed within the framework are effective in model based test suite generation and optimizatio
Automatically Discovering, Reporting and Reproducing Android Application Crashes
Mobile developers face unique challenges when detecting and reporting crashes
in apps due to their prevailing GUI event-driven nature and additional sources
of inputs (e.g., sensor readings). To support developers in these tasks, we
introduce a novel, automated approach called CRASHSCOPE. This tool explores a
given Android app using systematic input generation, according to several
strategies informed by static and dynamic analyses, with the intrinsic goal of
triggering crashes. When a crash is detected, CRASHSCOPE generates an augmented
crash report containing screenshots, detailed crash reproduction steps, the
captured exception stack trace, and a fully replayable script that
automatically reproduces the crash on a target device(s). We evaluated
CRASHSCOPE's effectiveness in discovering crashes as compared to five
state-of-the-art Android input generation tools on 61 applications. The results
demonstrate that CRASHSCOPE performs about as well as current tools for
detecting crashes and provides more detailed fault information. Additionally,
in a study analyzing eight real-world Android app crashes, we found that
CRASHSCOPE's reports are easily readable and allow for reliable reproduction of
crashes by presenting more explicit information than human written reports.Comment: 12 pages, in Proceedings of 9th IEEE International Conference on
Software Testing, Verification and Validation (ICST'16), Chicago, IL, April
10-15, 2016, pp. 33-4
Automated Test Input Generation for Android: Are We There Yet?
Mobile applications, often simply called "apps", are increasingly widespread,
and we use them daily to perform a number of activities. Like all software,
apps must be adequately tested to gain confidence that they behave correctly.
Therefore, in recent years, researchers and practitioners alike have begun to
investigate ways to automate apps testing. In particular, because of Android's
open source nature and its large share of the market, a great deal of research
has been performed on input generation techniques for apps that run on the
Android operating systems. At this point in time, there are in fact a number of
such techniques in the literature, which differ in the way they generate
inputs, the strategy they use to explore the behavior of the app under test,
and the specific heuristics they use. To better understand the strengths and
weaknesses of these existing approaches, and get general insight on ways they
could be made more effective, in this paper we perform a thorough comparison of
the main existing test input generation tools for Android. In our comparison,
we evaluate the effectiveness of these tools, and their corresponding
techniques, according to four metrics: code coverage, ability to detect faults,
ability to work on multiple platforms, and ease of use. Our results provide a
clear picture of the state of the art in input generation for Android apps and
identify future research directions that, if suitably investigated, could lead
to more effective and efficient testing tools for Android
Recommended from our members
The Effectiveness of <i>t</i>-Way Test Data Generation
Modern society is increasingly dependent on the correct functioning of software and increasingly so in areas that are considered safety related or safety critical. Therefore, there is an increasing need to be able to verify and validate that the software is in fact correct and will perform its intended function. Many approaches to this problem have been proposed; however, none seems likely to supplant the role of testing in the near future.
If we accept that there is, and will be, a continuing need to be able to test software then the question becomes one of how can this be done effectively, both in terms of ability to detect errors and in terms of cost. One avenue of research that offers prospects of improving both of these aspects is the automatic generation of test data.
There has recently been a large amount of work conducted in this area. One particularly promising direction has been the application of ideas from the field of experimental design and in particular, the field of t-way adequate factorial designs.
The area however, is not without issues; there is evidence that the technique is capable of detecting errors but that evidence is not unequivocal. Moreover, as with almost all work in the area of automatic test generation, there has been very little comparative work comparing the technique with other test data generation techniques. Worse, there has been effectively no work done that compares any automatic test data generation technique with the effectiveness of tests generated by humans. Another major issue with the technique is the number of tests that applying the technique can result in. This implies that there is a need for an automated oracle if the technique is to be successfully applied. The flaw with this is of course that in most situations the oracle is the human that is conducting the tests, a point often ignored in testing research.
The work presented here addresses both of these points. To do this I have used a code base taken from an industrial engine control system that has an existing set of high quality unit tests developed by hand. To complement this, several other techniques for automatically generating test data have been applied, namely random testing, random experimental designs and a technique for generating single factor experiments. To address the issue of being able to compare the error detection ability of all of the sets of test vectors, rather than the usual effectiveness surrogates of code coverage I have used mutation analysis on the code base to directly measure the ability of each set of test vectors to discover common coding errors. The results presented here show that test data generation techniques based on t-way factorial designs are at least as effective as handgenerated tests and superior to random testing and the factor experimental technique.
The oracle problem associated with the factorial design techniques was addressed using a test set minimisation approach. The mutation tool monitored which vectors could “kill” which code mutants. After a subset of the test vectors had been run, the most effective vectors were retained and the rest discarded. Likewise, mutants that were killed were removed from further consideration and the process repeated. Experimental results show that this minimisation procedure is effective at reducing computational overhead and is capable of producing final sets of test vectors that are comparable in size with the sets of hand-generated tests and so amenable to final hand checking
Combining Static and Dynamic Analysis for Vulnerability Detection
In this paper, we present a hybrid approach for buffer overflow detection in
C code. The approach makes use of static and dynamic analysis of the
application under investigation. The static part consists in calculating taint
dependency sequences (TDS) between user controlled inputs and vulnerable
statements. This process is akin to program slice of interest to calculate
tainted data- and control-flow path which exhibits the dependence between
tainted program inputs and vulnerable statements in the code. The dynamic part
consists of executing the program along TDSs to trigger the vulnerability by
generating suitable inputs. We use genetic algorithm to generate inputs. We
propose a fitness function that approximates the program behavior (control
flow) based on the frequencies of the statements along TDSs. This runtime
aspect makes the approach faster and accurate. We provide experimental results
on the Verisec benchmark to validate our approach.Comment: There are 15 pages with 1 figur
Telecommunications Network Planning and Maintenance
Telecommunications network operators are on a constant challenge to provide new services which require ubiquitous broadband access. In an attempt to do so, they are faced with many problems such as the network coverage or providing the guaranteed Quality of Service (QoS). Network planning is a multi-objective optimization problem which involves clustering the area of interest by minimizing a cost function which includes relevant parameters, such as installation cost, distance between user and base station, supported traffic, quality of received signal, etc. On the other hand, service assurance deals with the disorders that occur in hardware or software of the managed network. This paper presents a large number of multicriteria techniques that have been developed to deal with different kinds of problems regarding network planning and service assurance. The state of the art presented will help the reader to develop a broader understanding of the problems in the domain
New techniques for functional testing of microprocessor based systems
Electronic devices may be affected by failures, for example due to physical defects. These defects may be introduced during the manufacturing process, as well as during the normal operating life of the device due to aging. How to detect all these defects is not a trivial task, especially in complex systems such as processor cores. Nevertheless, safety-critical applications do not tolerate failures, this is the reason why testing such devices is needed so to guarantee a correct behavior at any time. Moreover, testing is a key parameter for assessing the quality of a manufactured product.
Consolidated testing techniques are based on special Design for Testability (DfT) features added in the original design to facilitate test effectiveness. Design, integration, and usage of the available DfT for testing purposes are fully supported by commercial EDA tools, hence approaches based on DfT are the standard solutions adopted by silicon vendors for testing their devices.
Tests exploiting the available DfT such as scan-chains manipulate the internal state of the system, differently to the normal functional mode, passing through unreachable configurations. Alternative solutions that do not violate such functional mode are defined as functional tests.
In microprocessor based systems, functional testing techniques include software-based self-test (SBST), i.e., a piece of software (referred to as test program) which is uploaded in the system available memory and executed, with the purpose of exciting a specific part of the system and observing the effects of possible defects affecting it. SBST has been widely-studies by the research community for years, but its adoption by the industry is quite recent.
My research activities have been mainly focused on the industrial perspective of SBST. The problem of providing an effective development flow and guidelines for integrating SBST in the available operating systems have been tackled and results have been provided on microprocessor based systems for the automotive domain. Remarkably, new algorithms have been also introduced with respect to state-of-the-art approaches, which can be systematically implemented to enrich SBST suites of test programs for modern microprocessor based systems. The proposed development flow and algorithms are being currently employed in real electronic control units for automotive products.
Moreover, a special hardware infrastructure purposely embedded in modern devices for interconnecting the numerous on-board instruments has been interest of my research as well. This solution is known as reconfigurable scan networks (RSNs) and its practical adoption is growing fast as new standards have been created. Test and diagnosis methodologies have been proposed targeting specific RSN features, aimed at checking whether the reconfigurability of such networks has not been corrupted by defects and, in this case, at identifying the defective elements of the network. The contribution of my work in this field has also been included in the first suite of public-domain benchmark networks
Generating Class-Level Integration Tests Using Call Site Information
Search-based approaches have been used in the literature to automate the
process of creating unit test cases. However, related work has shown that
generated unit-tests with high code coverage could be ineffective, i.e., they
may not detect all faults or kill all injected mutants. In this paper, we
propose CLING, an integration-level test case generation approach that exploits
how a pair of classes, the caller and the callee, interact with each other
through method calls. In particular, CLING generates integration-level test
cases that maximize the Coupled Branches Criterion (CBC). Coupled branches are
pairs of branches containing a branch of the caller and a branch of the callee
such that an integration test that exercises the former also exercises the
latter. CBC is a novel integration-level coverage criterion, measuring the
degree to which a test suite exercises the interactions between a caller and
its callee classes. We implemented CLING and evaluated the approach on 140
pairs of classes from five different open-source Java projects. Our results
show that (1) CLING generates test suites with high CBC coverage, thanks to the
definition of the test suite generation as a many-objectives problem where each
couple of branches is an independent objective; (2) such generated suites
trigger different class interactions and can kill on average 7.7% (with a
maximum of 50%) of mutants that are not detected by tests generated at the unit
level; (3) CLING can detect integration faults coming from wrong assumptions
about the usage of the callee class (32 for our subject systems) that remain
undetected when using automatically generated unit-level test suites
- …