Test Generation and Evaluation from High-Level Properties for Common Criteria Evaluations - The TASCCC Testing Tool

International audienceIn this paper, we present a model-based testing tool resulting from a research project, named TASCCC. This tool is a complete tool chain dedicated to property-based testing in UML/OCL, that integrates various technologies inside a dedicated Eclipse plug-in. The test properties are expressed in a dedicated language based on property patterns. These properties are then used for two purposes. First, they can be employed to evaluate the relevance of a test suite according to specific coverage criteria. Second, it is possible to generate test scenarios that will illustrate or exercise the property. These test scenarios are then unfolded and animated on the Smartesting's CertifyIt model animator, that is used to filter out infeasible sequences. This tool has been used in industrial partnership, aiming at providing an assistance for Common Criteria evaluations, especially by providing test generation reports used to show the link between the test cases and the Common Criteria artefacts

Génération automatique de scénarios de tests à partir de propriétés temporelles et de modèles comportementaux

In the work presented in this thesis, supported by the ANR TASCCC project, we propose a test generation technique using behavioral models in UML/OCL and temporal properties. To this end, we describe a temporal property langage based from the property patterns introduced by Dwyer et al. in [DAC99]. A property is a combination of a scope, representing the considered execution pathes in the system, and a pattern, a property that has to be satisfied inside the scope. We then give to each scope and pattern a specific automata-based semantics, called substitution automata. By combining a scope automaton with a pattern automaton, we obtain an automaton representing the semantics of the property. Next, we described nominal coverage criteria, based from classical coverage criteria over automata, specific to our property automata. These criteria focus on informations from the originating property, such as the property events, its scope and its pattern. We complemented this approach with another criterion that, by mutations over events held by certains transitions, aims at activating potentially faulty executions of the system by provoking forbidden events in the automata. We then described for each criterion an algorithm that aims at extracting pathes in the automata with respect to the considered criterion. These pathes are then translated to scenarios in an ad hoc langage that we defined. Finally, an unfolding process over these scenarios, with the help of driving commands embedded in the scenarios, allows the generation of abstract test cases. Finally, our approach has been validated on the study case presented in this document and on GlobalPlatform, an industrial-sized study case in the TASCCC project.Les travaux proposés dans cette thèse, effectuée dans le cadre du projet ANR TASCCC, présentent une technique de génération de tests à partir de modèles comportementaux en UML/OCL et de proprié- tés temporelles. Pour cela, nous décrivons un langage de propriétés temporelles inspiré des patrons de propriétés introduits par M. Dwyer et al.. Une propriété est définie comme la combinaison d'une portée, qui représente les exécutions du système dans laquelle un motif doit être satisfait. Nous associons à chaque portée et motif une sémantique à base d'automates particuliers, les automates de substitution. Par un mécanisme de substitution d'un automate de motif dans un automate de portée, nous obtenons un automate représentant la sémantique de la propriété. Nous avons ensuite défini des critères de couverture nominaux, inspirés des critères de couverture classiques sur les automates, spécifiques à nos automates de propriété. Ces critères se concentrent sur les informations supplémentaires apportées par la propriété originale, telles que ses évènements, sa portée et son motif. En complément, nous avons défini un critère de couverture qui, par le biais de mutation d'évènements de certaines transitions, permet de cibler des exécutions potentiellement dangereuses du système en tentant de provoquer les évènements interdits de la propriété. Ensuite, nous avons défini pour chaque critère un algorithme qui permet d'extraire des chemins dans l'automate, ciblant les éléments du critère considéré. Ces chemins sont traduits en scénarios dans un langage que nous avons défini. Enfin, un dépliage combinatoire de ces scénarios, éventuellement guidé par des directives de pilotage intégrées à celui-ci, permet la génération de cas de tests abstraits. Finalement, cette approche a été validée par une expérimentation sur une étude de cas dans ce document et sur GlobalPlatform, l'étude de cas de taille industrielle dans le cadre du projet TASCCC

Scenario Based Test Generation Using Test Designer

International audienceThis paper presents a Scenario Based Testing approach for UML/OCL behavioral models. Scenarios are expressed using a regular expression syntax, that makes it possible to specify iterations and choices between sequences of operation calls, specific operation behaviors to be activated, and intermediate states that have to be reached by the scenario. This expressive scenario language describes sequences of operations that compose the scenario and restrict the execution of the model to representative subset that complements the functional test cases produced by a test generator. This approach is tool supported by a scenario editor and coupled with the model animation engine of the Test Designer tool that is used to animate unfolded test cases and check their consistency with respect to the initial test scenario. The resulting abstract test cases can then be integrated back to the Test Designer repository so as to benefit from the test publishers offered by the tool

Scenario-based testing from UML/OCL behavioral models Application to POSIX compliance

International audienceWe present in this article a way to produce test suites applied to the POSIX mini-challenge based on a behavioral model of a file system manager written in UML/OCL. We illustrate the limitations of a fully automated test generation approach, which justifies the use of test scenarios as a complement to a functional testing approach. Scenarios are expressed through regular expressions describing sequences of operations, possibly punctuated by intermediate states that have to be reached by the execution of the model. Scenarios are unfolded into extended sequences of operations that are played on the model using symbolic animation techniques. We experimented our approach by testing the conformance of two different file systems w.r.t. the POSIX standard: a recent Linux distribution and a customized Java implementation of POSIX used to evaluate the relevance of our approach and its complementarity with a structural test generation approach

Show Me New Counterexamples: A Path-Based Approach

International audienceWe consider lightweight usage of model-checking for the debugging of Simulink models. A problem is that model-checkers typically return only one counterexample, which may slow down the debugging process. We propose an approach and a tool to produce several counterexamples, exemplifying different property violation patterns for a given version of the design. The approach uses data collected during the replay of the counterexamples to synthesize queries for the model-checker, so that it finds counterexamples that activate new paths. The approach is applied to an academic example and an industrial model from the automotive domain

Outil logiciel HAZOP-UML - Document des cas d'utilisation

Ce rapport contient la spécification de l'outil HAZOP-UML développé au LAAS-CNRS

Coverage Criteria for Model-Based Testing using Property Patterns

International audienc

Burst c-VEP Based BCI: Optimizing stimulus design for enhanced classification with minimal calibration data and improved user experience

The utilization of aperiodic flickering visual stimuli under the form of code-modulated Visual Evoked Potentials (c-VEP) represents a pivotal advancement in the field of reactive Brain–Computer Interface (rBCI). A major advantage of the c-VEP approach is that the training of the model is independent of the number and complexity of targets, which helps reduce calibration time. Nevertheless, the existing designs of c-VEP stimuli can be further improved in terms of visual user experience but also to achieve a higher signal-to-noise ratio, while shortening the selection time and calibration process. In this study, we introduce an innovative variant of code-VEP, referred to as “Burst c-VEP”. This original approach involves the presentation of short bursts of aperiodic visual flashes at a deliberately slow rate, typically ranging from two to four flashes per second. The rationale behind this design is to leverage the sensitivity of the primary visual cortex to transient changes in low-level stimuli features to reliably elicit distinctive series of visual evoked potentials. In comparison to other types of faster-paced code sequences, burst c-VEP exhibit favorable properties to achieve high bitwise decoding performance using convolutional neural networks (CNN), which yields potential to attain faster selection time with the need for less calibration data. Furthermore, our investigation focuses on reducing the perceptual saliency of c-VEP through the attenuation of visual stimuli contrast and intensity to significantly improve users’ visual comfort. The proposed solutions were tested through an offline 4-classes c-VEP protocol involving 12 participants. Following a factorial design, participants were instructed to focus on c-VEP targets whose pattern (burst and maximum-length sequences) and amplitude (100% or 40% amplitude depth modulations) were manipulated across experimental conditions. Firstly, the full amplitude burst c-VEP sequences exhibited higher accuracy, ranging from 90.5% (with 17.6s of calibration data) to 95.6% (with 52.8s of calibration data), compared to its m-sequence counterpart (71.4% to 85.0%). The mean selection time for both types of codes (1.5 s) compared favorably to reports from previous studies. Secondly, our findings revealed that lowering the intensity of the stimuli only slightly decreased the accuracy of the burst code sequences to 94.2% while leading to substantial improvements in terms of user experience. Taken together, these results demonstrate the high potential of the proposed burst codes to advance reactive BCI both in terms of performance and usability. The collected dataset, along with the proposed CNN architecture implementation, are shared through open-access repositories

Measuring Test Properties Coverage for evaluating UML/OCL Model-Based Tests

International audienceWe propose in the paper a test property specification language, dedicated to UML/OCL models. This language is intended to express temporal properties on the executions of the system, that one wants to test. It is based on patterns, specifying the behaviours one wants to exhibit/avoid, and scopes, defining the piece of execution trace on which a given pattern applies. Each property is a combination of a scope and a pattern, providing a means for a validation engineer to easily express temporal properties on a system, without using complex formal notations. Properties have the semantics of an event-based transition system whose coverage can be measured so as to evaluate the relevance of a given test suite. These principles aim at being used in the context of a research project, in which the security properties are expressed on an industrial case study of a smart card operating system. This approach makes it possible to assist the Common Criteria evaluation of the testing phase, that requires evidences of the extensiveness of the testing phase of a security product

A Compositional Automata-Based Semantics for Property Patterns

Print ISBN : 978-3-642-38612-1International audienceDwyer et al. define a language to specify dynamic properties based on predefined patterns and scopes. To define a property, the user has to choose a pattern and a scope among a limited number of them. Dwyer et al. define the semantics of these properties by translating each composition of a pattern and a scope into usual temporal logics (LTL, CTL, etc.). First, this translational semantics is not compositional and thus not easily extensible to other patterns/scopes. Second, it is not always faithful to the natural semantics of the informal definitions. In this paper, we propose a compositional automata-based approach defining the semantics of each pattern and each scope by an automaton. Then, we propose a composition operation in such a way that the property semantics is defined by composing the automata. Hence, the semantics is compositional and easily extensible as we show it by handling many extensions to the Dwyer et al.'s language. We compare our compositional semantics with the Dwyer et al.'s translational semantics by checking whether our automata are equivalent to the Büchi automata of the LTL expressions given by Dwyer et al. In some cases, our semantics reveals a lack of homogeneity within Dwyer et al.'s semantics