7,659 research outputs found
Is the Stack Distance Between Test Case and Method Correlated With Test Effectiveness?
Mutation testing is a means to assess the effectiveness of a test suite and
its outcome is considered more meaningful than code coverage metrics. However,
despite several optimizations, mutation testing requires a significant
computational effort and has not been widely adopted in industry. Therefore, we
study in this paper whether test effectiveness can be approximated using a more
light-weight approach. We hypothesize that a test case is more likely to detect
faults in methods that are close to the test case on the call stack than in
methods that the test case accesses indirectly through many other methods.
Based on this hypothesis, we propose the minimal stack distance between test
case and method as a new test measure, which expresses how close any test case
comes to a given method, and study its correlation with test effectiveness. We
conducted an empirical study with 21 open-source projects, which comprise in
total 1.8 million LOC, and show that a correlation exists between stack
distance and test effectiveness. The correlation reaches a strength up to 0.58.
We further show that a classifier using the minimal stack distance along with
additional easily computable measures can predict the mutation testing result
of a method with 92.9% precision and 93.4% recall. Hence, such a classifier can
be taken into consideration as a light-weight alternative to mutation testing
or as a preceding, less costly step to that.Comment: EASE 201
Will My Tests Tell Me If I Break This Code?
Automated tests play an important role in software evolution because they can
rapidly detect faults introduced during changes. In practice, code-coverage
metrics are often used as criteria to evaluate the effectiveness of test suites
with focus on regression faults. However, code coverage only expresses which
portion of a system has been executed by tests, but not how effective the tests
actually are in detecting regression faults. Our goal was to evaluate the
validity of code coverage as a measure for test effectiveness. To do so, we
conducted an empirical study in which we applied an extreme mutation testing
approach to analyze the tests of open-source projects written in Java. We
assessed the ratio of pseudo-tested methods (those tested in a way such that
faults would not be detected) to all covered methods and judged their impact on
the software project. The results show that the ratio of pseudo-tested methods
is acceptable for unit tests but not for system tests (that execute large
portions of the whole system). Therefore, we conclude that the coverage metric
is only a valid effectiveness indicator for unit tests.Comment: 7 pages, 3 figure
Mutation testing on an object-oriented framework: An experience report
This is the preprint version of the article - Copyright @ 2011 ElsevierContext
The increasing presence of Object-Oriented (OO) programs in industrial systems is progressively drawing the attention of mutation researchers toward this paradigm. However, while the number of research contributions in this topic is plentiful, the number of empirical results is still marginal and mostly provided by researchers rather than practitioners.
Objective
This article reports our experience using mutation testing to measure the effectiveness of an automated test data generator from a user perspective.
Method
In our study, we applied both traditional and class-level mutation operators to FaMa, an open source Java framework currently being used for research and commercial purposes. We also compared and contrasted our results with the data obtained from some motivating faults found in the literature and two real tools for the analysis of feature models, FaMa and SPLOT.
Results
Our results are summarized in a number of lessons learned supporting previous isolated results as well as new findings that hopefully will motivate further research in the field.
Conclusion
We conclude that mutation testing is an effective and affordable technique to measure the effectiveness of test mechanisms in OO systems. We found, however, several practical limitations in current tool support that should be addressed to facilitate the work of testers. We also missed specific techniques and tools to apply mutation testing at the system level.This work has been partially supported by the European Commission (FEDER) and Spanish Government under CICYT Project SETI (TIN2009-07366) and the Andalusian Government Projects ISABEL (TIC-2533) and THEOS (TIC-5906)
Putting formal specifications under the magnifying glass: Model-based testing for validation
A software development process is effectively an abstract form of model transformation, starting from an end-user model of requirements, through to a system model for which code can be automatically generated. The success (or failure) of such a transformation depends substantially on obtaining a correct, well-formed initial model that captures user concerns.
Model-based testing automates black box testing based on the model of the system under analysis. This paper proposes and evaluates a novel model-based testing technique that aims to reveal specification/requirement-related errors by generating test cases from a test model and exercising them on the design model. The case study outlined in the paper shows that a separate test model not only increases the level of objectivity of the requirements, but also supports the validation of the system under test through test case generation. The results obtained from the case study support the hypothesis that there may be discrepancies between the formal specification of the system modeled at developer end and the problem to be solved, and using solely formal verification methods may not be sufficient to reveal these. The approach presented in this paper aims at providing means to obtain greater confidence in the design model that is used as the basis for code generation
A framework for deadlock detection in core ABS
We present a framework for statically detecting deadlocks in a concurrent
object-oriented language with asynchronous method calls and cooperative
scheduling of method activations. Since this language features recursion and
dynamic resource creation, deadlock detection is extremely complex and
state-of-the-art solutions either give imprecise answers or do not scale. In
order to augment precision and scalability we propose a modular framework that
allows several techniques to be combined. The basic component of the framework
is a front-end inference algorithm that extracts abstract behavioural
descriptions of methods, called contracts, which retain resource dependency
information. This component is integrated with a number of possible different
back-ends that analyse contracts and derive deadlock information. As a
proof-of-concept, we discuss two such back-ends: (i) an evaluator that computes
a fixpoint semantics and (ii) an evaluator using abstract model checking.Comment: Software and Systems Modeling, Springer Verlag, 201
A Framework for Genetic Algorithms Based on Hadoop
Genetic Algorithms (GAs) are powerful metaheuristic techniques mostly used in
many real-world applications. The sequential execution of GAs requires
considerable computational power both in time and resources. Nevertheless, GAs
are naturally parallel and accessing a parallel platform such as Cloud is easy
and cheap. Apache Hadoop is one of the common services that can be used for
parallel applications. However, using Hadoop to develop a parallel version of
GAs is not simple without facing its inner workings. Even though some
sequential frameworks for GAs already exist, there is no framework supporting
the development of GA applications that can be executed in parallel. In this
paper is described a framework for parallel GAs on the Hadoop platform,
following the paradigm of MapReduce. The main purpose of this framework is to
allow the user to focus on the aspects of GA that are specific to the problem
to be addressed, being sure that this task is going to be correctly executed on
the Cloud with a good performance. The framework has been also exploited to
develop an application for Feature Subset Selection problem. A preliminary
analysis of the performance of the developed GA application has been performed
using three datasets and shown very promising performance
- …