Automatic instantiation of abstract tests on specific configurations for large critical control systems

Computer-based control systems have grown in size, complexity, distribution and criticality. In this paper a methodology is presented to perform an abstract testing of such large control systems in an efficient way: an abstract test is specified directly from system functional requirements and has to be instantiated in more test runs to cover a specific configuration, comprising any number of control entities (sensors, actuators and logic processes). Such a process is usually performed by hand for each installation of the control system, requiring a considerable time effort and being an error prone verification activity. To automate a safe passage from abstract tests, related to the so called generic software application, to any specific installation, an algorithm is provided, starting from a reference architecture and a state-based behavioural model of the control software. The presented approach has been applied to a railway interlocking system, demonstrating its feasibility and effectiveness in several years of testing experience

Toward a validation process for model based safety analysis

Today, Model Based processes become more and more widespread to achieve the analysis of a system. However, there is no formal testing approach to ensure that the formal model is compliant with the real system. In the paper, we choose to study AltaRica model. We present a general process to well construct and validate an AltaRica formal model. The focus is made on this validation phase, i.e. verifying the compliance between the model and the real system. For it, the proposed process recommends to build a specification for the AltaRica model. Then, the validation process is transformed to a classical verification problem between an implementation and a specification

Toward a validation process for model based safety analysis

International audienceToday, Model Based Safety Analysis processes become more and more widespread to achieve the safety analysis of a system. However and at our knowledge, there is no formal testing approach to ensure that the formal model is compliant with the real system. In the paper, we choose to study AltaRica model. We present a general process to well construct and validate an AltaRica formal model. The focus is made on this validation phase, i.e. verifying the compliance between the model and the real system. For it, the proposed process recommends to build a specification for the AltaRica model. Then, the validation process is transformed to a classical verification problem between an implementation and a specification. We present the first phase of a method to verify the compliance between the model and the specification

Canonical finite state machines for distributed systems

There has been much interest in testing from finite state machines (FSMs) as a result of their suitability for modelling or specifying state-based systems. Where there are multiple ports/interfaces a multi-port FSM is used and in testing, a tester is placed at each port. If the testers cannot communicate with one another directly and there is no global clock then we are testing in the distributed test architecture. It is known that the use of the distributed test architecture can affect the power of testing and recent work has characterised this in terms of local s-equivalence: in the distributed test architecture we can distinguish two FSMs, such as an implementation and a specification, if and only if they are not locally s-equivalent. However, there may be many FSMs that are locally s-equivalent to a given FSM and the nature of these FSMs has not been explored. This paper examines the set of FSMs that are locally s-equivalent to a given FSM M. It shows that there is a unique smallest FSM χmin(M) and a unique largest FSM χmax(M) that are locally s-equivalent to M. Here smallest and largest refer to the set of traces defined by an FSM and thus to its semantics. We also show that for a given FSM M the set of FSMs that are locally s-equivalent to M defines a bounded lattice. Finally, we define an FSM that, amongst all FSMs locally s-equivalent to M, has fewest states. We thus give three alternative canonical FSMs that are locally s-equivalent to an FSM M: one that defines the smallest set of traces, one that defines the largest set of traces, and one with fewest states. All three provide valuable information and the first two can be produced in time that is polynomial in terms of the number of states of M. We prove that the problem of finding an s-equivalent FSM with fewest states is NP-hard in general but can be solved in polynomial time for the special case where there are two ports

Testing a deterministic implementation against a non-controllable non-deterministic stream X-machine

A stream X-machine is a type of extended finite state machine with an associated development approach that consists of building a system from a set of trusted components. One of the great benefits of using stream X-machines for the purpose of specification is the existence of test generation techniques that produce test suites that are guaranteed to determine correctness as long as certain well-defined conditions hold. One of the conditions that is traditionally assumed to hold is controllability: this insists that all paths through the stream X-machine are feasible. This restrictive condition has recently been weakened for testing from a deterministic stream X-machine. This paper shows how controllability can be replaced by a weaker condition when testing a deterministic system against a non-deterministic stream X-machine. This paper therefore develops a new, more general, test generation algorithm for testing from a non-deterministic stream X-machine

Testing conformance of a deterministic implementation against a non-deterministic stream X-machine

Stream X-machines are a formalisation of extended finite state machines that have been used to specify systems. One of the great benefits of using stream X-machines, for the purpose of specification, is the associated test generation technique which produces a test that is guaranteed to determine correctness under certain design for test conditions. This test generation algorithm has recently been extended to the case where the specification is non-deterministic. However, the algorithms for testing from a non-deterministic stream X-machine currently have limitations: either they test for equivalence, rather than conformance or they restrict the source of non-determinism allowed in the specification. This paper introduces a new test generation algorithm that overcomes both of these limitations, for situations where the implementation is known to be deterministic

Test cases generation for functional tests of user interfaces

[ES] Tanto las aplicaciones software como las web se están haciendo cada vez más y más grandes necesitando así una mayor cantidad de recursos, por lo tanto la creación de estos requiere mucho tiempo para su desarrollo. Además de los requisitos de ingeniería, las diferentes herramientas de pruebas nos pueden ayudar a racionalizar y orientar el desarrollo de una aplicación a un buen final, con el objetivo de comprobar de nuevo si parece estar completa a través de secuencias de prueba para asegurarse de que la implementación del protocolo se ajusta a su especificación. La generación de datos a partir de tests automáticos ayuda a los testers a validar el software contra los requerimientos del usuario con mayor facilidad y verificar si el software está funcionando adecuadamente siguiendo los requisitos software. Este proceso representa más o menos el 50% del ciclo de vida del software. El enfoque más exhaustivo es poner a prueba todas las combinaciones posibles con el objetivo de descubrir algún error, pero esto con frecuencia es imposible debido a la gran cantidad de pruebas que supera con creces el tiempo y los recursos disponibles para ejecutarlos, por lo tanto, la parte crucial de las pruebas software consiste en seleccionar los datos que serán posteriormente analizados en dichas pruebas software. Los testers tienen que decidir qué datos de prueba deben utilizar, y que técnica heurística se necesita para resolver un problema dado automáticamente para reducir el número de pruebas combinatorias mientras se mantiene la capacidad de detección de fallos. La investigación en el campo de pruebas software en los últimos años se ha centrado en la modificación de métodos existentes, incluyendo entre ellos el más importante y en el que se ha basado más trabajo, el método W, a partir del cual se han desarrollado otros métodos como HIS, Wp, Mp y UIOv. En esta tesis se ha analizado el comportamiento de las técnicas existentes, tanto del método W como del Wp en nuestro propio statechart en busca de las limitaciones y mejoras. Como una alternativa a los métodos existentes, hemos propuesto un algoritmo basado en los métodos W y Wp, ideas propias e ideas recogidas de las distintas formas de trabajar con transiciones de un statechart dado. Aplicamos las técnicas de pruebas de caja gris (grey-box testing) que combina las pruebas de caja negra y blanca (black-box and white-box testing). Las pruebas de caja negra son una técnica software de testing en las que la funcionalidad del software se pone a prueba sin tener en cuenta la estructura del código interno (los testers prepararan los datos de entrada y los resultados esperados), mientras que las pruebas de caja blanca son una técnica de verificación de software que los ingenieros pueden utilizar para examinar si su código funciona como se esperaba (pruebas de estructura internas). El concepto de las pruebas de caja gris es simple, se basa en la realización de pruebas de caja negra basadas en casos de prueba realizados por personas que conocen el código del programa. Una de las mejoras que aporta nuestro método es que funciona de forma incremental reduciendo así la longitud de la secuencia de prueba generada por lo que este siempre empieza desde el mismo estado inicial de un FSM dado. Uno de los mayores problemas que encontramos es la aparición de bucles infinitos cuando aplicamos nuestra fórmula, que hemos resuelto mediante la adición de un número finito de iteraciones para el algoritmo de manera que no entre en un bucle infinito. Otra mejora es que se ha modificado el statechart inicial para identificar cada estado por separado, suponiendo que los casos de prueba pueden ser identificados y no es necesario aplicar la discriminación por conjuntos para ello. Nuestro objetivo es desarrollar un algoritmo que reduzca significativamente la duración de las secuencias de prueba necesarias para las pruebas de conformidad, manteniendo la misma capacidad de detección de fallos. Para demostrarlo, vamos a trabajar en el mismo ejemplo, con diversas modificaciones, donde se aplican diferentes métodos de prueba, aparte del que hemos desarrollado.[EN] Applications software and Web applications are getting bigger and need more resources, thus the creation of these requires a long time for its development. Apart from the Engineering Requirements, different testing tools can help us to streamline and to guide the development of an application to a good end, so as to check it again seems to be complete through test sequences to ensure that a protocol implementation conforms to its specification. Automatic test data generation helps testers to validate software against user requirements more easily and verify whether the software is working properly following software requirements. It accounts more or less the 50% of software life cycle. The most thorough approach is to test all possible combinations with the objective of discover some error, but this is quite often impossible due to the large number of tests which far exceeds the time and resources available to execute them, therefore, the crucial part of software testing is to select the test data for testing software. Testers have to decide which data test they should use, and a heuristic technique is needed to solve this problem automatically to reduce the number of combinatorial tests while maintaining the fault-detection capability of combinatorial testing. Research in Software testing in recent years has focused on modifying existing methods, including the most important and where more work is based on, the W method, from which other approaches have been developed as HIS, Wp, Mp and UIOv. This thesis has analyzed the behavior of existing techniques such as W and Wp method in our own statechart looking for limitations and improvements. As an alternative to existing methods, we have proposed an algorithm based on W and Wp method, own ideas, and collected ideas on different ways of working with statechart transitions. We apply the techniques of grey-box testing which combines black-box and white-box testing. Black-box testing is a software testing techniques in which functionality of the software under test (SUT) is tested without looking at the internal code structure (testers prepare test input and expected output) while white-box testing is a verification technique software that engineers can use to examine if their code works as expected (tests internal structures). The concept of grey-box testing is simple, is based in performing black box testing based on test cases performed by people who know the program inside. Some of the improvement is that the method works incrementally to reduce the length of generated test sequence so our new method always starts from the same starting state of the given FSM. This overcomes the problem that an extra leading sequenced may have to be added in the case that the test sequence generated started from a state different from the starting state of the given FSM. One of the biggest problems that we found was falling into infinite loops when we apply our formula by the appearance of them in our statechart, which we have solved by adding a finite number of iterations to the algorithm so that it does not end in an infinite loop. Another improvement is that we have modified the initial statechart to identify each state separately, assuming that test cases can be identified and do not need to apply discrimination set for this. Our goal is to develop an algorithm that reduces significantly the length of the test sequences required for conformance testing while maintaining the same fault detection capability. To prove it, we will work on the same example, with various modifications, where we will apply different testing methods apart from that one we have developed[CA] Tant les aplicacions software com les web s'estan fent cada vegada més i més grans necessitant així una major quantitat de recursos, per tant la creació d'aquests requereix molt de temps per al seu desenvolupament. A més dels requisits d'enginyeria, les diferents eines de proves ens poden ajudar a racionalitzar i orientar el desenvolupament d'una aplicació a un bona fi, amb l'objectiu de comprovar de nou si sembla estar completa a través de seqüències de prova per assegurar-se que la implementació del protocol s'ajusta a la seva especificació. La generació de dades a partir de tests automàtics ajuda als testers a validar el software contra els requeriments de l¿usuari amb més facilitat i verificar si el software està funcionant adequadament seguint els requisits. Aquest procés representa més o menys el 50% del cicle de vida del software. L'enfocament més exhaustiu és posar a prova totes les combinacions possibles amb l'objectiu de descobrir algun error, però això a sovint és impossible a causa de la gran quantitat de proves que supera amb creixes el temps i els recursos disponibles per executar-los, per tant, la part crucial de les proves software consisteix a seleccionar les dades que seran posteriorment analitzades en aquestes proves. Els testers han de decidir quines dades de prova han d'utilitzar, i que tècnica heurística es necessita per resoldre un problema donat automàticament per reduir el nombre de proves combinatòries mentre es manté la capacitat de detecció d'errors. La investigació en el camp de proves software en els últims anys s'ha centrat en la modificació de mètodes existents, incloent-hi el més important i en el qual s'ha basat més treball, el mètode W, a partir del qual s'han desenvolupat altres mètodes com HIS, Wp, Mp i UIOv. En aquesta tesi s'ha analitzat el comportament de les tècniques existents, tant del mètode W com del Wp en el nostre propi statechart a la recerca de les limitacions i millores. Com una alternativa als mètodes existents, hem proposat un algoritme basat en els mètodes W i Wp, idees pròpies i idees recollides de les diferents formes de treballar amb transicions d'un statechart donat. Apliquem les tècniques de proves de caixa gris (grey-box testing) que combina les proves de caixa negra i blanca (black-box and white-box testing). Les proves de caixa negra són una tècnica software de testing on la funcionalitat del software es posa a prova sense tenir en compte l'estructura del codi intern (els testers prepararan les dades d'entrada i els resultats esperats), mentre que les proves de caixa blanca són una tècnica de verificació de software que els enginyers poden utilitzar per examinar si el seu codi funciona com s'esperava (proves d'estructura internes). El concepte de les proves de caixa gris és simple, es basa en la realització de proves de caixa negra basades en casos de prova realitzats per persones que coneixen el codi del programa. Una de les millores que aporta el nostre mètode és que funciona de manera incremental reduint així la longitud de la seqüència de prova generada de manera que aquest sempre comença des del mateix estat inicial d'un FSM donat. Un dels majors problemes que trobem és l'aparició de bucles infinits quan apliquem la nostra fórmula, que hem resolt mitjançant l'addició d'un nombre finit d'iteracions per l'algoritme de manera que no entri en un bucle infinit. Una altra millora és que s'ha modificat el statechart inicial per identificar cada estat per separat, suposant que els casos de prova poden ser identificats i no cal aplicar la discriminació per conjunts per a això. El nostre objectiu és desenvolupar un algoritme que redueixi significativament la durada de les seqüències de prova necessàries per a les proves de conformitat, mantenint la mateixa capacitat de detecció d'errors. Per demostrar-ho, treballarem amb el mateix exemple, amb diverses modificacions, on s'apliquen diferents mètodes de testing, a part del que hem desenvolupatGramuntell Desco, F. (2012). Test cases generation for functional tests of user interfaces. http://hdl.handle.net/10251/3024

Design Time Methodology for the Formal Modeling and Verification of Smart Environments

Smart Environments (SmE) are intelligent and complex due to smart connectivity and interaction of heterogeneous devices achieved by complicated and sophisticated computing algorithms. Based on their domotic and industrial applications, SmE system may be critical in terms of correctness, reliability, safety, security and other such vital factors. To achieve error-free and requirement-compliant implementation of these systems, it is advisable to enforce a design process that may guarantee these factors by adopting formal models and formal verification techniques at design time. The e-Lite research group at Politecnico di Torino is developing solutions for SmE based on integration of commercially available home automation technologies with an intelligent ecosystem based on a central OSGi-based gateway, and distributed collaboration of intelligent applications, with the help of semantic web technologies and applications. The main goal of my research is to study new methodologies which are used for the modeling and verification of SmE. This goal includes the development of a formal methodology which ensures the reliable implementation of the requirements on SmE, by modeling and verifying each component (users, devices, control algorithms and environment/context) and the interaction among them, especially at various stages in design time, so that all the complexities and ambiguities can be reduced