10,768 research outputs found
Test-aware combinatorial interaction testing
Combinatorial interaction testing (CIT) approaches system- atically sample a given configuration space and select a set of configurations, in which each valid t-way option setting combination appears at least once. A battery of test cases are then executed in the selected configurations. Exist- ing CIT approaches, however, do not provide a system- atic way of handling test-specific inter-option constraints. Improper handling of such constraints, on the other hand, causes masking effects, which in turn causes testers to de- velop false confidence in their test processes, believing them have tested certain option setting combinations, when they in fact have not. In this work, to avoid the harmful conse- quences of masking effects caused by improper handling of test-specific constraints, we compute t-way test-aware cov- ering arrays. A t-way test-aware covering array is not just a set of configurations as is the case in traditional covering arrays, but a set of configurations, each of which is asso- ciated with a set of test cases. We furthermore present a set of empirical studies conducted by using two widely-used highly-configurable software systems as our subject applica- tions, demonstrating that test-specific constraints are likely to occur in practice and the proposed approach is a promis- ing and effective way of handling them
Fuzzy Adaptive Tuning of a Particle Swarm Optimization Algorithm for Variable-Strength Combinatorial Test Suite Generation
Combinatorial interaction testing is an important software testing technique
that has seen lots of recent interest. It can reduce the number of test cases
needed by considering interactions between combinations of input parameters.
Empirical evidence shows that it effectively detects faults, in particular, for
highly configurable software systems. In real-world software testing, the input
variables may vary in how strongly they interact, variable strength
combinatorial interaction testing (VS-CIT) can exploit this for higher
effectiveness. The generation of variable strength test suites is a
non-deterministic polynomial-time (NP) hard computational problem
\cite{BestounKamalFuzzy2017}. Research has shown that stochastic
population-based algorithms such as particle swarm optimization (PSO) can be
efficient compared to alternatives for VS-CIT problems. Nevertheless, they
require detailed control for the exploitation and exploration trade-off to
avoid premature convergence (i.e. being trapped in local optima) as well as to
enhance the solution diversity. Here, we present a new variant of PSO based on
Mamdani fuzzy inference system
\cite{Camastra2015,TSAKIRIDIS2017257,KHOSRAVANIAN2016280}, to permit adaptive
selection of its global and local search operations. We detail the design of
this combined algorithm and evaluate it through experiments on multiple
synthetic and benchmark problems. We conclude that fuzzy adaptive selection of
global and local search operations is, at least, feasible as it performs only
second-best to a discrete variant of PSO, called DPSO. Concerning obtaining the
best mean test suite size, the fuzzy adaptation even outperforms DPSO
occasionally. We discuss the reasons behind this performance and outline
relevant areas of future work.Comment: 21 page
Artificial table testing dynamically adaptive systems
Dynamically Adaptive Systems (DAS) are systems that modify their behavior and
structure in response to changes in their surrounding environment. Critical
mission systems increasingly incorporate adaptation and response to the
environment; examples include disaster relief and space exploration systems.
These systems can be decomposed in two parts: the adaptation policy that
specifies how the system must react according to the environmental changes and
the set of possible variants to reconfigure the system. A major challenge for
testing these systems is the combinatorial explosions of variants and
envi-ronment conditions to which the system must react. In this paper we focus
on testing the adaption policy and propose a strategy for the selection of
envi-ronmental variations that can reveal faults in the policy. Artificial
Shaking Table Testing (ASTT) is a strategy inspired by shaking table testing
(STT), a technique widely used in civil engineering to evaluate building's
structural re-sistance to seismic events. ASTT makes use of artificial
earthquakes that simu-late violent changes in the environmental conditions and
stresses the system adaptation capability. We model the generation of
artificial earthquakes as a search problem in which the goal is to optimize
different types of envi-ronmental variations
A Survey of Constrained Combinatorial Testing
Combinatorial Testing (CT) is a potentially powerful testing technique,
whereas its failure revealing ability might be dramatically reduced if it fails
to handle constraints in an adequate and efficient manner. To ensure the wider
applicability of CT in the presence of constrained problem domains, large and
diverse efforts have been invested towards the techniques and applications of
constrained combinatorial testing. In this paper, we provide a comprehensive
survey of representations, influences, and techniques that pertain to
constraints in CT, covering 129 papers published between 1987 and 2018. This
survey not only categorises the various constraint handling techniques, but
also reviews comparatively less well-studied, yet potentially important,
constraint identification and maintenance techniques. Since real-world programs
are usually constrained, this survey can be of interest to researchers and
practitioners who are looking to use and study constrained combinatorial
testing techniques
Bridging Data-Driven and Knowledge-Driven Approaches for Safety-Critical Scenario Generation in Automated Vehicle Validation
Automated driving vehicles~(ADV) promise to enhance driving efficiency and
safety, yet they face intricate challenges in safety-critical scenarios. As a
result, validating ADV within generated safety-critical scenarios is essential
for both development and performance evaluations. This paper investigates the
complexities of employing two major scenario-generation solutions: data-driven
and knowledge-driven methods. Data-driven methods derive scenarios from
recorded datasets, efficiently generating scenarios by altering the existing
behavior or trajectories of traffic participants but often falling short in
considering ADV perception; knowledge-driven methods provide effective coverage
through expert-designed rules, but they may lead to inefficiency in generating
safety-critical scenarios within that coverage. To overcome these challenges,
we introduce BridgeGen, a safety-critical scenario generation framework,
designed to bridge the benefits of both methodologies. Specifically, by
utilizing ontology-based techniques, BridgeGen models the five scenario layers
in the operational design domain (ODD) from knowledge-driven methods, ensuring
broad coverage, and incorporating data-driven strategies to efficiently
generate safety-critical scenarios. An optimized scenario generation toolkit is
developed within BridgeGen. This expedites the crafting of safety-critical
scenarios through a combination of traditional optimization and reinforcement
learning schemes. Extensive experiments conducted using Carla simulator
demonstrate the effectiveness of BridgeGen in generating diverse
safety-critical scenarios
- …