A formal evaluation of data flow path selection criteria

A number of path selection criteria have been proposed throughout the years. Unfortunately, little work has been done on comparing these criteria. To determine what would be an effective path selection criterion for revealing errors in programs, we have undertaken an evaluation of these criteria. This paper reports on the results of our evaluation of path selection criteria based on data flow relationships. We show how these criteria relate to each other, thereby demonstrating some of their strengths and weaknesses. In addition, we suggest minor changes to some criteria that improve their performance. We conclude with a discussion of the major limitations of these criteria and directions for future research

ASSINATURA DO TESTE ESTRUTURAL (ATE) - MÉTRICA BASEADA EM TESTE FLUXO DE DADOS E ANÁLISE DE MUTANTES SIGNATURE OF THE STRUCTURAL TEST (SST) - METRIC BASED ON DATA FLOW TEST AND MUTANT ANALYSIS

The structural test makes it possible to guarantee the quality of the software  by  analyzing  the  source  code.  In  this  sense,  this  paper presents a new method of structural testing with the application of data   flow   testing   and   mutant   analysis   techniques.   It   is   an arithmetic  model  called  Signature  of  the  Structural  Test  (SSt), which  receives parameters such as variables, operands, operators and   commands,   determines   a   metric   with   the   objective   of detecting   possible   semantic   and   logical   errors   between   the definitions of variables and their uses, such as also, reducing the test paths in the data graph and the mutants generated. The result was favorable to the continuation of the research with other testsand guidance for the construction of an appropriate tool.O teste estrutural possibilita garantir a qualidade do software pela análise do código fonte. Nesse sentido, este trabalho apresenta um novo método de teste estrutural com a aplicação das técnicas teste fluxo de dados e análise de mutantes. Trata-se de um modelo aritmético intitulado Assinatura do Teste  Estrutural  (AtE),  que recebe parâmetros como variáveis, operandos, operadores e comandos, determina  uma métrica com objetivos de  detectar possíveis erros semânticos e lógicos entre  as definições de variáveis e seus usos, como também, reduzir os caminhos de testes no grafo de dados e os mutantes gerados. O resultado mostrou-se favorável à continuação da pesquisa com outros ensaios edirecionamento para construção de ferramenta apropriada

Dynamic data flow testing

Data flow testing is a particular form of testing that identifies data flow relations as test objectives. Data flow testing has recently attracted new interest in the context of testing object oriented systems, since data flow information is well suited to capture relations among the object states, and can thus provide useful information for testing method interactions. Unfortunately, classic data flow testing, which is based on static analysis of the source code, fails to identify many important data flow relations due to the dynamic nature of object oriented systems. This thesis presents Dynamic Data Flow Testing, a technique which rethinks data flow testing to suit the testing of modern object oriented software. Dynamic Data Flow Testing stems from empirical evidence that we collect on the limits of classic data flow testing techniques. We investigate such limits by means of Dynamic Data Flow Analysis, a dynamic implementation of data flow analysis that computes sound data flow information on program traces. We compare data flow information collected with static analysis of the code with information observed dynamically on execution traces, and empirically observe that the data flow information computed with classic analysis of the source code misses a significant part of information that corresponds to relevant behaviors that shall be tested. In view of these results, we propose Dynamic Data Flow Testing. The technique promotes the synergies between dynamic analysis, static reasoning and test case generation for automatically extending a test suite with test cases that execute the complex state based interactions between objects. Dynamic Data Flow Testing computes precise data flow information of the program with Dynamic Data Flow Analysis, processes the dynamic information to infer new test objectives, which Dynamic Data Flow Testing uses to generate new test cases. The test cases generated by Dynamic Data Flow Testing exercise relevant behaviors that are otherwise missed by both the original test suite and test suites that satisfy classic data flow criteria

Improving fault coverage and minimising the cost of fault identification when testing from finite state machines

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.Software needs to be adequately tested in order to increase the confidence that the system being developed is reliable. However, testing is a complicated and expensive process. Formal specification based models such as finite state machines have been widely used in system modelling and testing. In this PhD thesis, we primarily investigate fault detection and identification when testing from finite state machines. The research in this thesis is mainly comprised of three topics - construction of multiple Unique Input/Output (UIO) sequences using Metaheuristic Optimisation Techniques (MOTs), the improved fault coverage by using robust Unique Input/Output Circuit (UIOC) sequences, and fault diagnosis when testing from finite state machines. In the studies of the construction of UIOs, a model is proposed where a fitness function is defined to guide the search for input sequences that are potentially UIOs. In the studies of the improved fault coverage, a new type of UIOCs is defined. Based upon the Rural Chinese Postman Algorithm (RCPA), a new approach is proposed for the construction of more robust test sequences. In the studies of fault diagnosis, heuristics are defined that attempt to lead to failures being observed in some shorter test sequences, which helps to reduce the cost of fault isolation and identification. The proposed approaches and techniques were evaluated with regard to a set of case studies, which provides experimental evidence for their efficacy.Brunel Research Initiative and Enterprise Fund (BRIEF) Award from Brunel University and Departmental bursary from Department of Information Systems and Computing, Brunel Universit

Structural testing techniques for the selective revalidation of software

The research in this thesis addresses the subject of regression testing. Emphasis is placed on developing a technique for selective revalidation which can be used during software maintenance to analyse and retest only those parts of the program affected by changes. In response to proposed program modifications, the technique assists the maintenance programmer in assessing the extent of the program alterations, in selecting a representative set of test cases to rerun, and in identifying any test cases in the test suite which are no longer required because of the program changes. The proposed technique involves the application of code analysis techniques and operations research. Code analysis techniques are described which derive information about the structure of a program and are used to determine the impact of any modifications on the existing program code. Methods adopted from operations research are then used to select an optimal set of regression tests and to identify any redundant test cases. These methods enable software, which has been validated using a variety of structural testing techniques, to be retested. The development of a prototype tool suite, which can be used to realise the technique for selective revalidation, is described. In particular, the interface between the prototype and existing regression testing tools is discussed. Moreover, the effectiveness of the technique is demonstrated by means of a case study and the results are compared with traditional regression testing strategies and other selective revalidation techniques described in this thesis

Improving fault coverage and minimising the cost of fault identification when testing from finite state machines

Software needs to be adequately tested in order to increase the confidence that the system being developed is reliable. However, testing is a complicated and expensive process. Formal specification based models such as finite state machines have been widely used in system modelling and testing. In this PhD thesis, we primarily investigate fault detection and identification when testing from finite state machines. The research in this thesis is mainly comprised of three topics - construction of multiple Unique Input/Output (UIO) sequences using Metaheuristic Optimisation Techniques (MOTs), the improved fault coverage by using robust Unique Input/Output Circuit (UIOC) sequences, and fault diagnosis when testing from finite state machines. In the studies of the construction of UIOs, a model is proposed where a fitness function is defined to guide the search for input sequences that are potentially UIOs. In the studies of the improved fault coverage, a new type of UIOCs is defined. Based upon the Rural Chinese Postman Algorithm (RCPA), a new approach is proposed for the construction of more robust test sequences. In the studies of fault diagnosis, heuristics are defined that attempt to lead to failures being observed in some shorter test sequences, which helps to reduce the cost of fault isolation and identification. The proposed approaches and techniques were evaluated with regard to a set of case studies, which provides experimental evidence for their efficacy.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Twenty-five Years Of Research In Structural And Mutation Testing

Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)Research in software testing has been carried out for approximately forty years, but its importance has escalated very quickly in the last ten or fifteen years. In particular, structural and mutation testing are techniques which havereceived a large amount of investment in both academia and software development industry. In this paper, we draw a historical perspective on how they appeared and how they evolved. In particular, the main contributions on structural and mutation testing from two Brazilian researching groups - ICMC-USP and FEEC-UNICAMP - are described. We highlight the workproduced and published in these twenty-five years in the Brazilian Symposium on Software Engineering and elsewhere, as well its impact in the community of software testing. © 2011 IEEE.4049FAPESP,CNPq,Google,Mack Pesquisa,CAPES,PRCEU-USPFundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)Harrold, M.J., Testing: A roadmap (2000) ICSE - Future of SE Track, pp. 61-72Beizer, B., (1984) Software System Testing and Quality Assurance, , NewYork: Van Nostrand Reinhold CompanyPezz, M., Young, M., (2007) Software Testing and Analysis: Process, Principles and Techniques, , John Wiley and SonsSpillner, A., Linz, T., Schaefer, H., (2007) Software Testing Foundation, , RockynookMaldonado, J.C., Chaim, M.L., Jino, M., Seleo de casos de testes baseada na anlise de fluxo de dados atravs dos critrios potenciais usos II SBES - Simpsio Brasileiro de Engenharia de Software (SBES 88), Canela, RS, Oct. 1988, pp. 24-35Martins, E., Integrando injecao de falhas e testes formais na validacao da tolerancia a falhas IX Simposio Brasileiro de Engenharia de Software (SBES 95), Recife, PE, 1995, pp. 223-239Rombaldi, A.M.A.P.J.B.S., Proteste: Um ambiente de apoio ao teste de programas VI Simpsio Brasileiro de Engenharia de Software (SBES 92), Canela, RS, Oct. 1992McCabe, T.J., A software complexity measure (1976) IEEE Transactions on Software Engineering, 2 (6), pp. 308-320. , DecMyers, G.J., (1979) The Art of Software Testing, , Wiley, New YorkBeizer, B., (1990) Software Testing Techniques, , 2nd ed. New York: Van Nostrand Reinhold CompanyHowden, W.E., Methodology for the generation of program test data (1975) IEEE Computer, C-24 (5), pp. 554-559. , MayHowden, W.E., (1987) Functional Program Testing and Analysis, , New York: McGrall-HillHecht, M.S., (1977) Flow Analysis of Computer Programs, , NewYork:North HollandHerman, P.M., A data flow analysis approach to program testing (1976) Australian Computer Journal, 8 (3). , NovRapps, S., Weyuker, E.J., SELECTING SOFTWARE TEST DATA USING DATA FLOW INFORMATION (1985) IEEE Transactions on Software Engineering, SE-11 (4), pp. 367-375Ntafos, S.C., On required element testing (1984) IEEE Transactions on Software Engineering, SE-10, pp. 795-803. , NovLaski, J.W., Korel, B., A data flow oriented program testing strategy (1983) IEEE Transactions on Software Engineering, 9 (3). , MayUral, H., Yang, B., A structural test selection criterion (1988) Information Processing Letters, 28, pp. 157-163Haley, A., Zweben, S., Development and application of a white box approach to integration testing (1984) The Journal of Systems and Software, 4, pp. 309-315Linnenkugel, U., Müllerburg, M., Test data selection criteria for (software) integration testing First International Conference on Systems Integration, Morristown, NJ, Apr. 1990, pp. 709-717Harrold, M.J., Soffa, M.L., Selecting and using data for integration test (1991) IEEE Software, 8 (2), pp. 58-65. , MarJin, Z., Offut, A.J., Integration testing based on software couplings X Annual Conference on Computer Assurance (COMPASS 95), Gaithersburg, Maryland, Jan. 1995, pp. 13-23Frankl, P.G., Weyuker, E.J., An applicable family of data flow testing criteria (1988) IEEE Transactions on Software Engineering, 14 (10), pp. 1483-1498. , OctMaldonado, J.C., Chaim, M.L., Jino, M., Potential uses testing criteria: A step towards bridging the gap in the presence of infeasible paths Proceedings of the XII International Conference of the Chilean Computer Society, 1992Maldonado, J.C., Chaim, M.L., Jino, M., Using the essential branch concept to support data-flow criteria application Proceedings of the V International Conference on Software Engineering and Its Applications, 1992Weyuker, E.J., The complexity of data flow for test data selection (1984) Information Processing Letters, 19 (2), pp. 103-109. , AugMaldonado, J.C., Verglio, S.R., Chaim, M.L., Jino, M., Critrios potenciais usos: Anlise da aplicao de um benchmark VI SBES -Simpsio Brasileiro de Engenharia de Software (SBES 92), Gramado, RS, Nov. 1992, pp. 357-374Vilela, P.R.S., (1998) Critrios Potenciais Usos de Integrao: Definio e Anlise, , Ph.D. dissertation, DCA/FEEC/UNICAMP, Campinas, SP, AprVergilio, S.R., Maldonado, J.C., Jino, M., Soares, I.W., Constraint based structural testing criteria (2006) Journal of Systems and Software, 79 (6), pp. 756-771Wong, W., Sugeta, T., Li, J.J., Maldonado, J.C., Coverage testing software architectural design in SDL (2003) Journal of Computer Networks, 42 (3), pp. 359-374. , JunEler, M.M., Delamaro, M.E., Masiero, P.C., Maldonado, J.C., Builtin structural testing of web services (2010) Proceedings of the Brazilian Symposium on Software Engineering, CBSOFT 2010, pp. 1-10Vincenzi, A.M.R., Maldonado, J.C., Wong, W.E., Delamaro, M.E., Coverage testing of Java programs and components (2005) Journal of Science of Computer ProgrammingVincenzi, A.M.R., Delamaro, M.E., Maldonado, J.C., Wong, W.E., Establishing structural testing criteria for java bytecode (2006) Softw., Pract. Exper., 36 (14), pp. 1513-1541Lemos, O.A.L., Franchin, I.G., Masiero, P.C., Integration testing of object-oriented and aspect-oriented programs: A structural pairwise approach for java (2009) Sci. Comput. Program., 74 (10), pp. 861-878Lemos, O.A.L., Vincenzi, A.M.R., Maldonado, J.C., Masiero, P.C., Control and data flow structural testing criteria for aspect-oriented programs (2007) Journal of Systems and Software, 80 (6), pp. 862-882Chaim, M.L., (1991) POKE-TOOL: A Tool to Support Data Flow Based Structural Test of Programs, , Master's thesis, DCA/FEEC/UNICAMP, Campinas, SP, Apr. (In Portuguese)Bueno, P.M.S., Jino, M., Automatic test data generation for program paths using genetic algorithms (2002) International Journal of Software Engineering and Knowledge Engineering, 12 (6), pp. 691-710Bueno, P.M., Jino, M., Wong, W.E., Diversity oriented test data generation using metaheuristic search techniques (2011) Information Sciences, , http://caocao.myipcn.org/science/article/pii/S0020025511000442, vol. In Press, Corrected Proof, pp. -, [Online]. AvailableChaim, M.L., Maldonado, J.C., Jino, M., A debugging strategy based on the requirements of testing (2004) Journal of Software Maintenance, 16 (4-5), pp. 277-308Crespo, A.N., Jino, M., Pasquini, A., Maldonado, J.C., A binomial software reliability model based on coverage of structural testing criteria (2008) Empirical Software Engineering, 13 (2), pp. 185-209. , DOI 10.1007/s10664-007-9055-3Pasquini, A., Crespo, A.N., Matrella, P., Sensitivity of reliability-growth models to operational profile errors vs testing accuracy (1996) IEEE Transactions on Reliability, 45 (4), pp. 531-540. , PII S0018952996095462Vincenzi, A.M.R., Wong, W.E., Delamaro, M.E., Maldonado, J.C., JaBUTi: A coverage analysis tool for Java programs XVII Brazilian Symposium on Software Engineering, Manaus, AM, Brasil, Oct. 2003, pp. 79-84Delamaro, M.E., Vincenzi, A.M.R., Maldonado, J.C., A strategy to perform coverage testing of mobile applications (2006) Proceedings of the 2006 International Workshop on Automation of Software Test, AST 2006, pp. 118-124Delamaro, M.E., Vincenzi, A.M.R., Structural Testing of Mobile Agents (2003) Lecture Notes on Computer Science, 2952, pp. 73-85. , International Workshop on Scientific Engineering of Java Distributed Applications (FIDJI 2003), ser. E. A. Nicolas Guelfi and G. Reggio, Eds. Springer, NovLi, J.J., Weiss, D.M., Yee, H., An automatically-generated run-time instrumenter to reduce coverage testing overhead (2008) Proceedings of the 3rd International Workshop on Automation of Software Test, AST 2008, pp. 49-56Soffa, M.L., Walcott, K.R., Mars, J., Exploiting hardware advances for software testing and debugging: Nier track (2011) Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, pp. 888-891Hutchins, M., Foster, H., Goradia, T., Ostrand, T.J., Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria (1994) ICSE, pp. 191-200Santelices, R.A., Jones, J.A., Yu, Y., Harrold, M.J., Lightweight fault-localization using multiple coverage types (2009) 31st International Conference on Software Engineering, ICSE 2009, pp. 56-66Misurda, J., Clause, J.A., Reed, J.L., Childers, B.R., Soffa, M.L., Demand-driven structural testing with dynamic instrumentation (2005) 27th International Conference on Software Engineering (ICSE 2005), pp. 156-165Santelices, R.A., Harrold, M.J., Efficiently monitoring data-flow test coverage (2007) 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2007), pp. 343-352DeMillo, R.A., Lipton, R.J., Sayward, F.G., Hints on test data selection: Help for the practicing programmer (1978) Computer, 11 (4), pp. 34-41. , AprilDeMillo, R., Guindi, D., McCracken, W., Offutt, A., King, K., An extended overview of the mothra software testing environment (1988) Software Testing, Verification, and Analysis, 1988., Proceedings of the Second Workshop on, pp. 142-151. , julAgrawal, H., DeMillo, R.A., Hathaway, R., Hsu, W., Hsu, W., Krauser, E.W., Martin, R.J., Spafford, E.H., (1989) Design of Mutant Operators for the C Programming Language, , Software Engineering Research Center, Purdue University, West Lafayette, IN, Tech. Rep. SERC-TR41-P, MarChoi, B., Mathur, A., Pattison, B., Pmothra: Scheduling mutants for execution on a hypercube (1989) SIGSOFT Softw. Eng. Notes, 14, pp. 58-65. , http://doi.acm.org/10.1145/75309.75316, November [Online]. AvailableOffutt, A.J., Pargas, R.P., Fichter, S.V., Khambekar, P.K., Mutation testing of software using a mimd computer (1992) 1992 International Conference on Parallel Processing, pp. 257-266Offutt, A.J., Rothermel, G., Zapf, C., An experimental evaluation of selective mutation (1993) ICSE '93, pp. 100-107. , http://portal.acm.org/citation.cfm?id=257572.257597, Proceedings of the 15th international conference on Software Engineering, ser. Los Alamitos, CA, USA: IEEE Computer Society Press, [Online]. AvailableOffutt, A., Pan, J., Detecting equivalent mutants and the feasible path problem (1996) Computer Assurance, 1996. COMPASS '96, 'Systems Integrity. Software Safety. Process Security'. Proceedings of the Eleventh Annual Conference on, pp. 224-236. , junOffutt, A.J., Craft, W.M., Using compiler optimization techniques to detect equivalent mutants (1994) Software Testing, Verification and Reliability, 4 (3), pp. 131-154. , http://dx.doi.org/10.1002/stvr.4370040303, [Online]. AvailableJia, Y., Harman, M., An analysis and survey of the development of mutation testing (2010) IEEE Transactions on Software Engineering, 99. , no. PrePrintsDelamaro, M.E., (1993) Proteum: Um Ambiente de Teste Baseado Na Análise de Mutantes (Proteum: A Testing Environment Based on Mutation Testing), , Master Thesis, SCE-ICMSC-USP, São Carlos - SP, OctDelamaro, M.E., Maldonado, J.C., Jino, M., Chaim, M.L., Proteum: A Testing Tool Based on Mutation Analysis Software Tools Seesion - VII Brazilian Symposium on Software Engineering, Rio de Janeiro - RJ, Oct. 1993, pp. 31-33Vincenzi, A.M.R., (1998) Subsídios para O Estabelecimento de Estratégias de Teste Baseadas Na Técnica de Mutaço Aid for Establishing Mutation Based Testing Strategies, , Master's thesis, ICMC-USP, So Carlos - SP, NovVincenzi, A.M.R., Maldonado, J.C., Barbosa, E.F., Delamaro, M.E., Unit and integration testing strategies for C programs using mutation-based criteria (2001) Software Testing, Verification and Reliability, 11 (4), pp. 249-268. , DecBarbosa, E.F., (1998) A Contribution for the Determination of A Sufficient Mutant Operators Set for C-program Testing, , Master's thesis, ICMCUSP, São Carlos, SP, Nov. (in Portuguese)Fabbri, S.C.P.F., Maldonado, J.C., Masiero, P.C., Delamaro, M.E., Mutation analysis testing for finite state machines (1994) 5th International Symposium on Software Reliability Engineering (ISSRE'94), pp. 220-229. , Monterey - CA: IEEE Computer Society Press, NovFabbri, S.C.P.F., Delamaro, M.E., Maldonado, J.C., Masieiro, P.C., Aplicaçã o da análise de mutantes na validac̃ã o de especificações baseadas em redes de petri (mutation analysis based on petri net models) VIII Brazilian Symposium on Software Engineering, Curitiba -PR, Brazil, Oct. 1994, pp. 423-437Fabbri, S.C.P.F., Maldonado, J.C., Sugeta, T., Masiero, P.C., Mutation testing applied to validate specifications based on statecharts ISSRE - International Symposium on Software Reliability Systems, Nov. 1999, pp. 210-219Simo, A.S., Maldonado, J.C., Fabbri, S.C.P.F., Proteum-RS/PN: A tool to support edition, simulation and validation of Petri nets based on mutation testing XIV Brazilian Symposium on Software Engineering, João Pessoa - PB - Brazil, Oct. 2000Delamaro, M.E., Maldonado, J.C., Mathur, A.P., Interface Mutation: An Approach for Integration Testing (2001) IEEE Transactions on Software Engineering, 27 (3), pp. 228-247. , MarAraujo, R.F., Maldonado, J.C., Delamaro, M.E., Devising Mutant Operators for Dynamic Systems Models by Applying the Hazop Study, , submitedAraujo, R.F., Maldonado, J.C., Delamaro, M.E., Tetoods Extension for Mutation Testing, , submite