Fault injection testing of software implemented fault tolerance mechanisms of distributed systems

PhD ThesisOne way of gaining confidence in the adequacy of fault tolerance mechanisms of a system is to test the system by injecting faults and see how the system performs under faulty conditions. This thesis investigates the issues of testing software-implemented fault tolerance mechanisms of distributed systems through fault injection. A fault injection method has been developed. The method requires that the target software system be structured as a collection of objects interacting via messages. This enables easy insertion of fault injection objects into the target system to emulate incorrect behaviour of faulty processors by manipulating messages. This approach allows one to inject specific classes of faults while not requiring any significant changes to the target system. The method differs from the previous work in that it exploits an object oriented approach of software implementation to support the injection of specific classes of faults at the system level. The proposed fault injection method has been applied to test software-implemented reliable node systems: a TMR (triple modular redundant) node and a fail-silent node. The nodes have integrated fault tolerance mechanisms and are expected to exhibit certain behaviour in the presence of a failure. The thesis describes how various such mechanisms (for example, clock synchronisation protocol, and atomic broadcast protocol) were tested. The testing revealed flaws in implementation that had not been discovered before, thereby demonstrating the usefulness of the method. Application of the approach to other distributed systems is also described in the thesis.CEC ESPRIT programme, UK Engineering and Physical Sciences Research Council (EPSRC)

Designing Fault Injection Experiments Using State-based Model To Test A Space Software

Software for space applications requires significant testing. This paper presents an evaluation of the CoFI testing methodology as applied to actual space software, where deterministic fault cases derived from state-based models were executed using the software-implemented fault injection technique. Different models were used to represent the behavior of embedded software in a real satellite computer under the presence of both normal inputs and external faults in communication, processor, and memory. CoFI methodology was used for model construction, the Condado tool for test derivation, and the QSEE-TAS tool for test execution. In total, 8,620% of 471 fault cases detected errors in the software; this is a very large number, and more so considering that the software had already been tested by the company which developed it before being subject the CoFI methodology. © Springer-Verlag Berlin Heidelberg 2007.4746 LNCS170178Ambrosio, A.M.: CoFI: uma abordagem combinando teste de conformidade e injeção de falhas para validação de software em aplicações espaciais. INPE-13264-TDI/1031. Instituto Nacional de Pesquisas Espaciais - INPE (2005)Ambrosio, A.M., Martins, E., Vijaykumar, N.L., Carvalho, S.V., A Methodology for Designing Fault Injection Experiments as an Addition to Communication Systems Conformance Testing (2005) Proceedings of the 1st Workshop on Dependable Software - Tools and Methods in the IEEE Conference on Dependable System and Network, , Yokohama, Japan, 28 June, 1 JulyArlat, J., Aguera, M., Amat, L., Crouzet, Y., Fabre, J.-C., Laprie, J.-C., Martins, E., Powell, D., Fault Injection for Dependability Validation: A Methodology and Some Applications (1990) IEEE Tr on SE, 16 (2), pp. 166-182Binder, R., (2000) Testing Object-Oriented Systems-Models, Patterns and Tools, , Addison-Wesley, ReadingCavalli, A., Gervy, C., Prokopenko, S.: New Approaches for Passive Testing using Extended Finite State Machine Specification. In: WTCS, Canada (2001)Chandra, R., Lefever, R.M., Cukier, M., Sanders, W.H.A: Global-state triggered fault injector for distributed system evaluation (2004) IEEE Transaction on Parallel and Distributed Systems, 15 (7), pp. 593-605Chow, T.S., Testing software design modeled by finite state machines (1978) IEEE Trans on Sw Engineering (TSE), 3, pp. 178-187Dssouli, H., Salek, K., Aboulhamid, E., En-Nouaary, A., Bourhfir, C., Test Development for Comm. Protocols: Towards Automation (1999) Computer Networks, 31, pp. 1835-1872Echtle, K., Chen, Y., Evaluation of Deterministic Fault Injection for Fault-Tolerant Protocol Testing (1991) IEEE 21th Annual International Symposium on Fault-Tolerant Computing, pp. 418-425. , Montreal, pp, IEEE Computer Society Press, Los Alamitoshttp://radhome.gsfc.nasa.gov/radhome/papers/seeca1.htm, Goddard Space Flight Center GSFC, accessed March 2007, available atHolzmann, G.J., (1990) Design and validation of computer protocols, , Prentice-Hall, Englewood Cliffs(1991), International Organization for Standardization ISO/IEC- IS9646 International standard conformance testing methodology and framework. GeneveMartins, E., Mattiello-Francisco, F.A., Tool for Fault Injection and Conformance Testing of Distributed Systems (2003) LNCS, 2847, pp. 282-302Martins, E., Sabião, S.B., Ambrosio, A.M., ConData: A Tool for Automating Specification-based Test Case Generation for Communication Systems (1999) Software Quality Journal, 8 (4), pp. 303-319Mattiello-Francisco, M.F., Santiago, V.A., Costa, R., Jogaib, L., Verificação e Validação na terceirização de software embarcado em aplicaçães espaciais (2006) Simpósio Brasiliero de Qualidade de Software - SBQS2006, Villa Velha, pp. 368-375. , ES, Brazil, ppSantiago, V., Mattiello-Francisco, F., Costa, R., Silva, W.P., Ambrosio, A.M., QSEE Project: An Experience in Outsourcing Software Development for Space Applications (2007) The Nineteenth International Conference on Software Engineering and Knowledge Engineering (SEKE'07), , Boston, EUASilva, W.P., QSEE-TAS: Uma Ferramenta para Execução e Relato Automatizados de Testes de Software para Aplicaçães Espaciais (2006) XX Brazilian Symposium on Software Engineering-SBE