Improved test quality using robust unique input/output circuit sequences (UIOCs)

In finite state machine (FSM) based testing, the problem of fault masking in the unique input/ output (UIO) sequence may degrade the test performance of the UIO based methods. This paper investigates this problem and proposes the use of a new type of unique input/output circuit (UIOC) sequence for state verification, which may help to overcome the drawbacks that exist in the UIO based techniques. When constructing a UIOC, overlap and internal state observation schema are used to increase the robustness of a test sequence. Test quality is compared by using the forward UIO method (F-method), the backward UIO method (B-method) and the UIOC method (C-method) separately. Robustness of the UIOCs constructed by the algorithm given in this paper is also compared with those constructed by the algorithm given previously. Experimental results suggest that the C-method outperforms the F- and the B-methods and the UIOCs constructed by the Algorithm given in this paper, are more robust than those constructed by other proposed algorithms

Using formal methods to support testing

Formal methods and testing are two important approaches that assist in the development of high quality software. While traditionally these approaches have been seen as rivals, in recent years a new consensus has developed in which they are seen as complementary. This article reviews the state of the art regarding ways in which the presence of a formal specification can be used to assist testing

Improving fault coverage and minimising the cost of fault identification when testing from finite state machines

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.Software needs to be adequately tested in order to increase the confidence that the system being developed is reliable. However, testing is a complicated and expensive process. Formal specification based models such as finite state machines have been widely used in system modelling and testing. In this PhD thesis, we primarily investigate fault detection and identification when testing from finite state machines. The research in this thesis is mainly comprised of three topics - construction of multiple Unique Input/Output (UIO) sequences using Metaheuristic Optimisation Techniques (MOTs), the improved fault coverage by using robust Unique Input/Output Circuit (UIOC) sequences, and fault diagnosis when testing from finite state machines. In the studies of the construction of UIOs, a model is proposed where a fitness function is defined to guide the search for input sequences that are potentially UIOs. In the studies of the improved fault coverage, a new type of UIOCs is defined. Based upon the Rural Chinese Postman Algorithm (RCPA), a new approach is proposed for the construction of more robust test sequences. In the studies of fault diagnosis, heuristics are defined that attempt to lead to failures being observed in some shorter test sequences, which helps to reduce the cost of fault isolation and identification. The proposed approaches and techniques were evaluated with regard to a set of case studies, which provides experimental evidence for their efficacy.Brunel Research Initiative and Enterprise Fund (BRIEF) Award from Brunel University and Departmental bursary from Department of Information Systems and Computing, Brunel Universit

A conformance test framework for the DeviceNet fieldbus

The DeviceNet fieldbus technology is introduced and discussed. DeviceNet is an open standard fieldbus which uses the proven Controller Area Network technology. As an open standard fieldbus, the device conformance is extremely important to ensure smooth operation. The error management in DeviceNet protocol is highlighted and an error injection technique is devised to test the implementation under test for the correct error-recovery conformance. The designed Error Frame Generator prototype allows the error management and recovery of DeviceNet implementations to be conformance tested. The Error Frame Generator can also be used in other Controller Area Network based protocols. In addition, an automated Conformance Test Engine framework has been defined for realising the conformance testing of DeviceNet implementations. Automated conformance test is used to achieve consistent and reliable test results, apart from the benefits in time and personnel savings. This involves the investigations and feasibility studies in adapting the ISO 9646 conformance test standards for use in DeviceNet fieldbus. The Unique Input/Output sequences method is used for the generation of DeviceNet conformance tests. The Unique Input/Output method does not require a fully specified protocol specification and gives shorter test sequences, since only specific state information is needed. As conformance testing addresses only the protocol verification, it is foreseen that formal method validation of the DeviceNet protocol must be performed at some stage to validate the DeviceNet specification

Improving fault coverage and minimising the cost of fault identification when testing from finite state machines

Software needs to be adequately tested in order to increase the confidence that the system being developed is reliable. However, testing is a complicated and expensive process. Formal specification based models such as finite state machines have been widely used in system modelling and testing. In this PhD thesis, we primarily investigate fault detection and identification when testing from finite state machines. The research in this thesis is mainly comprised of three topics - construction of multiple Unique Input/Output (UIO) sequences using Metaheuristic Optimisation Techniques (MOTs), the improved fault coverage by using robust Unique Input/Output Circuit (UIOC) sequences, and fault diagnosis when testing from finite state machines. In the studies of the construction of UIOs, a model is proposed where a fitness function is defined to guide the search for input sequences that are potentially UIOs. In the studies of the improved fault coverage, a new type of UIOCs is defined. Based upon the Rural Chinese Postman Algorithm (RCPA), a new approach is proposed for the construction of more robust test sequences. In the studies of fault diagnosis, heuristics are defined that attempt to lead to failures being observed in some shorter test sequences, which helps to reduce the cost of fault isolation and identification. The proposed approaches and techniques were evaluated with regard to a set of case studies, which provides experimental evidence for their efficacy.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Test cases generation for functional tests of user interfaces

[ES] Tanto las aplicaciones software como las web se están haciendo cada vez más y más grandes necesitando así una mayor cantidad de recursos, por lo tanto la creación de estos requiere mucho tiempo para su desarrollo. Además de los requisitos de ingeniería, las diferentes herramientas de pruebas nos pueden ayudar a racionalizar y orientar el desarrollo de una aplicación a un buen final, con el objetivo de comprobar de nuevo si parece estar completa a través de secuencias de prueba para asegurarse de que la implementación del protocolo se ajusta a su especificación. La generación de datos a partir de tests automáticos ayuda a los testers a validar el software contra los requerimientos del usuario con mayor facilidad y verificar si el software está funcionando adecuadamente siguiendo los requisitos software. Este proceso representa más o menos el 50% del ciclo de vida del software. El enfoque más exhaustivo es poner a prueba todas las combinaciones posibles con el objetivo de descubrir algún error, pero esto con frecuencia es imposible debido a la gran cantidad de pruebas que supera con creces el tiempo y los recursos disponibles para ejecutarlos, por lo tanto, la parte crucial de las pruebas software consiste en seleccionar los datos que serán posteriormente analizados en dichas pruebas software. Los testers tienen que decidir qué datos de prueba deben utilizar, y que técnica heurística se necesita para resolver un problema dado automáticamente para reducir el número de pruebas combinatorias mientras se mantiene la capacidad de detección de fallos. La investigación en el campo de pruebas software en los últimos años se ha centrado en la modificación de métodos existentes, incluyendo entre ellos el más importante y en el que se ha basado más trabajo, el método W, a partir del cual se han desarrollado otros métodos como HIS, Wp, Mp y UIOv. En esta tesis se ha analizado el comportamiento de las técnicas existentes, tanto del método W como del Wp en nuestro propio statechart en busca de las limitaciones y mejoras. Como una alternativa a los métodos existentes, hemos propuesto un algoritmo basado en los métodos W y Wp, ideas propias e ideas recogidas de las distintas formas de trabajar con transiciones de un statechart dado. Aplicamos las técnicas de pruebas de caja gris (grey-box testing) que combina las pruebas de caja negra y blanca (black-box and white-box testing). Las pruebas de caja negra son una técnica software de testing en las que la funcionalidad del software se pone a prueba sin tener en cuenta la estructura del código interno (los testers prepararan los datos de entrada y los resultados esperados), mientras que las pruebas de caja blanca son una técnica de verificación de software que los ingenieros pueden utilizar para examinar si su código funciona como se esperaba (pruebas de estructura internas). El concepto de las pruebas de caja gris es simple, se basa en la realización de pruebas de caja negra basadas en casos de prueba realizados por personas que conocen el código del programa. Una de las mejoras que aporta nuestro método es que funciona de forma incremental reduciendo así la longitud de la secuencia de prueba generada por lo que este siempre empieza desde el mismo estado inicial de un FSM dado. Uno de los mayores problemas que encontramos es la aparición de bucles infinitos cuando aplicamos nuestra fórmula, que hemos resuelto mediante la adición de un número finito de iteraciones para el algoritmo de manera que no entre en un bucle infinito. Otra mejora es que se ha modificado el statechart inicial para identificar cada estado por separado, suponiendo que los casos de prueba pueden ser identificados y no es necesario aplicar la discriminación por conjuntos para ello. Nuestro objetivo es desarrollar un algoritmo que reduzca significativamente la duración de las secuencias de prueba necesarias para las pruebas de conformidad, manteniendo la misma capacidad de detección de fallos. Para demostrarlo, vamos a trabajar en el mismo ejemplo, con diversas modificaciones, donde se aplican diferentes métodos de prueba, aparte del que hemos desarrollado.[EN] Applications software and Web applications are getting bigger and need more resources, thus the creation of these requires a long time for its development. Apart from the Engineering Requirements, different testing tools can help us to streamline and to guide the development of an application to a good end, so as to check it again seems to be complete through test sequences to ensure that a protocol implementation conforms to its specification. Automatic test data generation helps testers to validate software against user requirements more easily and verify whether the software is working properly following software requirements. It accounts more or less the 50% of software life cycle. The most thorough approach is to test all possible combinations with the objective of discover some error, but this is quite often impossible due to the large number of tests which far exceeds the time and resources available to execute them, therefore, the crucial part of software testing is to select the test data for testing software. Testers have to decide which data test they should use, and a heuristic technique is needed to solve this problem automatically to reduce the number of combinatorial tests while maintaining the fault-detection capability of combinatorial testing. Research in Software testing in recent years has focused on modifying existing methods, including the most important and where more work is based on, the W method, from which other approaches have been developed as HIS, Wp, Mp and UIOv. This thesis has analyzed the behavior of existing techniques such as W and Wp method in our own statechart looking for limitations and improvements. As an alternative to existing methods, we have proposed an algorithm based on W and Wp method, own ideas, and collected ideas on different ways of working with statechart transitions. We apply the techniques of grey-box testing which combines black-box and white-box testing. Black-box testing is a software testing techniques in which functionality of the software under test (SUT) is tested without looking at the internal code structure (testers prepare test input and expected output) while white-box testing is a verification technique software that engineers can use to examine if their code works as expected (tests internal structures). The concept of grey-box testing is simple, is based in performing black box testing based on test cases performed by people who know the program inside. Some of the improvement is that the method works incrementally to reduce the length of generated test sequence so our new method always starts from the same starting state of the given FSM. This overcomes the problem that an extra leading sequenced may have to be added in the case that the test sequence generated started from a state different from the starting state of the given FSM. One of the biggest problems that we found was falling into infinite loops when we apply our formula by the appearance of them in our statechart, which we have solved by adding a finite number of iterations to the algorithm so that it does not end in an infinite loop. Another improvement is that we have modified the initial statechart to identify each state separately, assuming that test cases can be identified and do not need to apply discrimination set for this. Our goal is to develop an algorithm that reduces significantly the length of the test sequences required for conformance testing while maintaining the same fault detection capability. To prove it, we will work on the same example, with various modifications, where we will apply different testing methods apart from that one we have developed[CA] Tant les aplicacions software com les web s'estan fent cada vegada més i més grans necessitant així una major quantitat de recursos, per tant la creació d'aquests requereix molt de temps per al seu desenvolupament. A més dels requisits d'enginyeria, les diferents eines de proves ens poden ajudar a racionalitzar i orientar el desenvolupament d'una aplicació a un bona fi, amb l'objectiu de comprovar de nou si sembla estar completa a través de seqüències de prova per assegurar-se que la implementació del protocol s'ajusta a la seva especificació. La generació de dades a partir de tests automàtics ajuda als testers a validar el software contra els requeriments de l¿usuari amb més facilitat i verificar si el software està funcionant adequadament seguint els requisits. Aquest procés representa més o menys el 50% del cicle de vida del software. L'enfocament més exhaustiu és posar a prova totes les combinacions possibles amb l'objectiu de descobrir algun error, però això a sovint és impossible a causa de la gran quantitat de proves que supera amb creixes el temps i els recursos disponibles per executar-los, per tant, la part crucial de les proves software consisteix a seleccionar les dades que seran posteriorment analitzades en aquestes proves. Els testers han de decidir quines dades de prova han d'utilitzar, i que tècnica heurística es necessita per resoldre un problema donat automàticament per reduir el nombre de proves combinatòries mentre es manté la capacitat de detecció d'errors. La investigació en el camp de proves software en els últims anys s'ha centrat en la modificació de mètodes existents, incloent-hi el més important i en el qual s'ha basat més treball, el mètode W, a partir del qual s'han desenvolupat altres mètodes com HIS, Wp, Mp i UIOv. En aquesta tesi s'ha analitzat el comportament de les tècniques existents, tant del mètode W com del Wp en el nostre propi statechart a la recerca de les limitacions i millores. Com una alternativa als mètodes existents, hem proposat un algoritme basat en els mètodes W i Wp, idees pròpies i idees recollides de les diferents formes de treballar amb transicions d'un statechart donat. Apliquem les tècniques de proves de caixa gris (grey-box testing) que combina les proves de caixa negra i blanca (black-box and white-box testing). Les proves de caixa negra són una tècnica software de testing on la funcionalitat del software es posa a prova sense tenir en compte l'estructura del codi intern (els testers prepararan les dades d'entrada i els resultats esperats), mentre que les proves de caixa blanca són una tècnica de verificació de software que els enginyers poden utilitzar per examinar si el seu codi funciona com s'esperava (proves d'estructura internes). El concepte de les proves de caixa gris és simple, es basa en la realització de proves de caixa negra basades en casos de prova realitzats per persones que coneixen el codi del programa. Una de les millores que aporta el nostre mètode és que funciona de manera incremental reduint així la longitud de la seqüència de prova generada de manera que aquest sempre comença des del mateix estat inicial d'un FSM donat. Un dels majors problemes que trobem és l'aparició de bucles infinits quan apliquem la nostra fórmula, que hem resolt mitjançant l'addició d'un nombre finit d'iteracions per l'algoritme de manera que no entri en un bucle infinit. Una altra millora és que s'ha modificat el statechart inicial per identificar cada estat per separat, suposant que els casos de prova poden ser identificats i no cal aplicar la discriminació per conjunts per a això. El nostre objectiu és desenvolupar un algoritme que redueixi significativament la durada de les seqüències de prova necessàries per a les proves de conformitat, mantenint la mateixa capacitat de detecció d'errors. Per demostrar-ho, treballarem amb el mateix exemple, amb diverses modificacions, on s'apliquen diferents mètodes de testing, a part del que hem desenvolupatGramuntell Desco, F. (2012). Test cases generation for functional tests of user interfaces. http://hdl.handle.net/10251/3024

Testing and Active Learning of Resettable Finite-State Machines

This thesis proposes novel active-learning algorithms and testing methods for deterministic finite-state machines that (i) have a specified transition from every state on each input of the (fixed) alphabet and (ii) can be reliably reset to the initial state on request. These algorithms rely on the novel methods of construction of separating sequences. Extensive evaluation demonstrates that the described testing and learning methods are the most efficient in terms of the amount of interaction by a tester with the system under test