A binomial software reliability model based on coverage of structural testing criteria

A new approach to software reliability modeling is discussed where variables indirectly related with software reliability are used to provide additional information for the modeling process. Previous studies, empirical and theoretical evidences, and results from experiments indicate that there is a strong relationship between software reliability and coverage of program elements required to be exercised by structural testing criteria. This paper develops a binomial type coverage-based software reliability model through the definition of a coverage-based failure rate function. The Binomial software reliability Model Based on Coverage-BMBC-is proposed and discussed. In the BMBC test data between failures is used instead of time as independent variable; the model was assessed with test data from a real application, making use of the following structural testing criteria: all-nodes, all-edges, and potential-uses-a data-flow based family of testing criteria. The results from our experiments have shown that our modeling approach has some advantages over some traditional reliability models and points to a very promising research direction in software reliability.13218520

Applying Code Coverage Approach To An Infinite Failure Software Reliability Model

An approach to software reliability modeling based on code coverage is used to derive the Infinite Failure software reliability Model Based on Code Coverage - IFMBC. Our aim was to verify the soundness of the approach under different assumptions. The IFMBC was assessed with test data from a real application, making use of the following structural testing criteria: all-nodes, all-edges, and potential-uses - a data-flow based family of criteria. The IFMBC was shown to be as good as the Geometric Model - GEO, found to be the best traditional time-based model that fits the data. Results from the analysis also show that the IFMBC is as good as the BMBC - Binomial software reliability Model Based on Coverage - a model previously derived using the code coverage approach, indicating it to be effective under different modeling assumptions. © 2009 IEEE.216226Musa, J.D., Ianino, A., Okumoto, K., (1987) Software Reliability - Measurement, Prediction, Application, , McGraw-Hill, New YorkHudson, A., Program Errors as a Birth and Death Process (1967) Technical Report SP - 3011, , Santa Monica, Cal, Systems Development CorporationJelinski, Z., Moranda, P.B., Software Reliability Research (1972) Proceedings of the Statistical Methods for the Evaluation of Computer System Performance, pp. 465-484. , Academic PressShooman, M.L., Probabilistic Models for Software Reliability Prediction (1972) Statistical Computer Performance Evaluation, pp. 485-502. , W. Freiberg, Ed. New York: Academic PressLyu, M.R., Handbook of Software Reliability Engineering (1996) IEEE Computer Society Press, , Los Alamitos, CAPrediction and Measurement of Software Reliability (1990) Dependability of Critical Computer Systems 3, , Bishop, P, ed, Elsevier Science Publisher, LondonButler, R.W., Finelli, G.B., The Unfeasibility of Experimental Quantification of Life-critical Software Reliability (1991) Proceedings of the ACM SIGSOFT '91 Conference on Software for Critical Systems, pp. 66-76. , ACM Press, DecemberHamlet, D., Are We testing for True Reliability (1992) IEEE Software, , IEEE Computer Society, JulyLittlewood, B., String, L., Validation of Ultrahigh Dependability for Software-based Systems (1993) Communications of the ACM, , ACM, JanVaradan, G.S., Trends in Reliability and Test Strategies (1995) IEEE Software, p. 10. , IEEE Computer Society, MayChen, M., Lyu, M.R., Wong, W.E., An Empirical Study of the Correlation between Code Coverage and Reliability Estimation (1996) Proceedings of the Third International Software Metrics Symposium, , Berlin, Germany, March 25-26Chen, M., Tools and Techniques for Testing Based Software Reliability Estimation, (1994), Ph. D. Dissertation, Purdue University, West Lafayette, Indiana, AugustChen, M., Mathur, A.P., Rego, V.J., Effect of Testing Technique on Software Reliability Estimates Obtained Using A Time-Domain Model (1995) IEEE Transactions on Reliability, pp. 97-103. , IEEE, MarchRamsey, J., Basili, V.R., Analyzing the Test Process Using Structural Coverage (1985) Proceedings ICSE, pp. 306-312. , 85, ppGarg, P., Investigating Coverage - Reliability Relationship and Sensitivity of Reliability to Errors in Operational Profile (1994), Technical Report, Department of Computer Sciences, Purdue University, West Lafayette, MayVeevers, A., Marshall, A., A Relationship Between Software Coverage Metrics and Reliability (1994) Software Testing, Verification and Reliability, pp. 3-8Malaiya, Y.K., Li, N., Bieman, J., Karcich, R., Skibe, B., The Relationship Between Test Coverage and Reliability (1994) Proceedings of the Fifth International Symposium on Software Reliability Engineering, pp. 186-195. , Monterey, CA, November 6-9Jalote, P., Muralidhara, Y.R., A Coverage Based Model for Software Reliability Estimation (1994) Proceedings of the First International Conference on Software Testing, Reliability and Quality Assurance, pp. 6-10. , New Delhi, ÍndiaHorgan, J.R., Mathur, A.P., Pasquini, A., Rego, V.J., Perils of Software Reliability Modeling (1995), Technical Report SERC-TR-160-P, Purdue UniversityMalaiya, Y.K., Li, N., Bieman, J., Karcich, R., Skibbe, B., Software Test Coverage and Reliability (1996), Technical Report CS-96-128, Colorado State UniversityGokhale, S.S., Philip, T., Marinos, P.N., Trivedi, K.S., Unification of Finite Failure Non-Homogeneous Poisson Process Models through Test Coverage (1996) Proceedings of The Seventh International Symposium on Software Reliability Engineering, pp. 289-299. , White Plains, NY, USAKrishnamurthy, S., Mathur, A.P., On Predicting Reliability of Modules Using Code Coverage (1996) Proceedings of the Conference of the Centre for Advanced Studies on Collaborative Research, pp. 1-12. , Toronto, CanadaChen, M., Lyu, M.R., Wong, W.E., Incorporating Code Coverage in the Reliability Estimation for Fault-Tolerant Software (1997) Proceedings of the Symposium on Reliable Distributed Systems, , Durham, NCChen, M., Lyu, M.R., Wong, W.E., Effect of Code Coverage on Software Reliability Measurement (2001) IEEE Transactions on Reliability, pp. 165-170Ye, R., Malaiya, Y.K., Relationship Between Test Effectiveness and Coverage (2002) Proceedings of the 13th International Symposium on Software Reliability Engineering - ISSREMalaiya, Y.K., Li, N., Bieman, J., Karcich, R., Software Reliability Growth with Test Coverage (2002) IEEE Transactions on Reliability, pp. 420-426Grottke, M., A Vector Markov Model for Structural Coverage Growth and the Number of Failure Occurrences (2002) Proceedings of the 13th International Symposium on Software Reliability Engineering - ISSREPham, H., Zhang, X., NHPP Software Reliability and Cost Models with Testing Coverage (2003) European Journal of Operational Research, 145, pp. 443-454Li, M., Wei, Y., Desovski, D., Nejad, H., Ghose, S., Cukic, B., Smidts, C., Validation of a Methodology for Assessing Software Reliability (2004) Proceedings of the 15th International Symposium on Software Reliability Engineering, pp. 66-76. , Saint-Malo, FranceFrate, F.D., Garg, P., Mathur, A.P., Pasquini, A., Experiments to Investigate the Correlation Between Code Coverage and Software Reliability (1995) SERC-TR-162-P, Software Engineering Research Center, , Purdue University, West Lafayette, Indiana 47907, AprilCrespo, A.N., Pasquini, A., Jino, M., Madonado, J.C., Cobertura dos Critérios Potenciais-usos e Confiabilidade do Software (1997) Anais do XI Simpósio Brasileiro de Engenharia de Software, , Fortaleza, in PortugueseCrespo, A.N., Pasquini, A., Jino, M., Madonado, J.C., A Binomial Software Reliability Model Based on Coverage of Structural Testing Criteria (2000) Anais do XI Simpósio Brasileiro de Engenharia de Software, , João Pessoa, PBCrespo, A.N., Pasquini, A., Jino, M., Madonado, J.C., A Binomial Software Reliability Model Based on Coverage of Structural Testing Criteria (2008) Empirical Software Engineering, Springer Netherlands, 13 (2), pp. 185-209. , AprilCai, K.Y., Towards a conceptual framework of software run reliability modeling (2000) Information Sciences, 126, pp. 137-163Rapps, S., Weyuker, E.J., Selecting Software Test Data Using Data Flow Information (1985) IEEE Transactions on Software Engineering, pp. 367-375Maldonado, J.C., Chaim, M.L., Jino, M., Bridging the Gap in the Presence of Infeasible Paths: Potential Uses Testing Criteria (1992) XII International Conference of the SCCC, pp. 323-340. , Sociedad Chilena de Ciencia de la Computacion, Santiago, ChileFrankl, P.G., Weyuker, E.J., An Applicable Family of Data Flow Testing Criteria (1988) IEEE Transactions on Software Engineering, pp. 1483-1498Crespo, A.N., Modelos de Confiabilidade de Software Baseados em Cobertura de Critérios Estruturais de Teste (1997) Tese de Doutorado, , DCA/FEEC/UNICAMP, Campinas, SP, in PortuguesePasquini, A., Crespo, A.N., Matrella, P., Sensitivity of Reliability Growth Models to Operational Profile Errors vs Testing Accuracy (1996) IEEE Transactions on Reliability, pp. 531-540Chaim, M.L., (1991) Uma Ferramenta para Suporte ao Teste Estrutural de Programas Baseado em Análise de Fluxo de Dados, , Dissertação de Mestrado, DCA/FEE/UNICAMP, Campinas, SP, in PortugueseNelson, F., Estimating Software Reliability From Test Data (1978) Microelectronics and Reliability, pp. 67-73Farr, W.H., Smith, O.D., Statistical Modeling and Estimation of Reliability Functions for Software (SMERFS) User's Guide (1993) Naval Surface Warfare Center, , VirginiaSiegel, S., (1956) Nonparametric Statistics for the Behavioral Sciences, , McGraw Hill Kogakush

Twenty-five Years Of Research In Structural And Mutation Testing

Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)Research in software testing has been carried out for approximately forty years, but its importance has escalated very quickly in the last ten or fifteen years. In particular, structural and mutation testing are techniques which havereceived a large amount of investment in both academia and software development industry. In this paper, we draw a historical perspective on how they appeared and how they evolved. In particular, the main contributions on structural and mutation testing from two Brazilian researching groups - ICMC-USP and FEEC-UNICAMP - are described. We highlight the workproduced and published in these twenty-five years in the Brazilian Symposium on Software Engineering and elsewhere, as well its impact in the community of software testing. © 2011 IEEE.4049FAPESP,CNPq,Google,Mack Pesquisa,CAPES,PRCEU-USPFundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)Harrold, M.J., Testing: A roadmap (2000) ICSE - Future of SE Track, pp. 61-72Beizer, B., (1984) Software System Testing and Quality Assurance, , NewYork: Van Nostrand Reinhold CompanyPezz, M., Young, M., (2007) Software Testing and Analysis: Process, Principles and Techniques, , John Wiley and SonsSpillner, A., Linz, T., Schaefer, H., (2007) Software Testing Foundation, , RockynookMaldonado, J.C., Chaim, M.L., Jino, M., Seleo de casos de testes baseada na anlise de fluxo de dados atravs dos critrios potenciais usos II SBES - Simpsio Brasileiro de Engenharia de Software (SBES 88), Canela, RS, Oct. 1988, pp. 24-35Martins, E., Integrando injecao de falhas e testes formais na validacao da tolerancia a falhas IX Simposio Brasileiro de Engenharia de Software (SBES 95), Recife, PE, 1995, pp. 223-239Rombaldi, A.M.A.P.J.B.S., Proteste: Um ambiente de apoio ao teste de programas VI Simpsio Brasileiro de Engenharia de Software (SBES 92), Canela, RS, Oct. 1992McCabe, T.J., A software complexity measure (1976) IEEE Transactions on Software Engineering, 2 (6), pp. 308-320. , DecMyers, G.J., (1979) The Art of Software Testing, , Wiley, New YorkBeizer, B., (1990) Software Testing Techniques, , 2nd ed. New York: Van Nostrand Reinhold CompanyHowden, W.E., Methodology for the generation of program test data (1975) IEEE Computer, C-24 (5), pp. 554-559. , MayHowden, W.E., (1987) Functional Program Testing and Analysis, , New York: McGrall-HillHecht, M.S., (1977) Flow Analysis of Computer Programs, , NewYork:North HollandHerman, P.M., A data flow analysis approach to program testing (1976) Australian Computer Journal, 8 (3). , NovRapps, S., Weyuker, E.J., SELECTING SOFTWARE TEST DATA USING DATA FLOW INFORMATION (1985) IEEE Transactions on Software Engineering, SE-11 (4), pp. 367-375Ntafos, S.C., On required element testing (1984) IEEE Transactions on Software Engineering, SE-10, pp. 795-803. , NovLaski, J.W., Korel, B., A data flow oriented program testing strategy (1983) IEEE Transactions on Software Engineering, 9 (3). , MayUral, H., Yang, B., A structural test selection criterion (1988) Information Processing Letters, 28, pp. 157-163Haley, A., Zweben, S., Development and application of a white box approach to integration testing (1984) The Journal of Systems and Software, 4, pp. 309-315Linnenkugel, U., Müllerburg, M., Test data selection criteria for (software) integration testing First International Conference on Systems Integration, Morristown, NJ, Apr. 1990, pp. 709-717Harrold, M.J., Soffa, M.L., Selecting and using data for integration test (1991) IEEE Software, 8 (2), pp. 58-65. , MarJin, Z., Offut, A.J., Integration testing based on software couplings X Annual Conference on Computer Assurance (COMPASS 95), Gaithersburg, Maryland, Jan. 1995, pp. 13-23Frankl, P.G., Weyuker, E.J., An applicable family of data flow testing criteria (1988) IEEE Transactions on Software Engineering, 14 (10), pp. 1483-1498. , OctMaldonado, J.C., Chaim, M.L., Jino, M., Potential uses testing criteria: A step towards bridging the gap in the presence of infeasible paths Proceedings of the XII International Conference of the Chilean Computer Society, 1992Maldonado, J.C., Chaim, M.L., Jino, M., Using the essential branch concept to support data-flow criteria application Proceedings of the V International Conference on Software Engineering and Its Applications, 1992Weyuker, E.J., The complexity of data flow for test data selection (1984) Information Processing Letters, 19 (2), pp. 103-109. , AugMaldonado, J.C., Verglio, S.R., Chaim, M.L., Jino, M., Critrios potenciais usos: Anlise da aplicao de um benchmark VI SBES -Simpsio Brasileiro de Engenharia de Software (SBES 92), Gramado, RS, Nov. 1992, pp. 357-374Vilela, P.R.S., (1998) Critrios Potenciais Usos de Integrao: Definio e Anlise, , Ph.D. dissertation, DCA/FEEC/UNICAMP, Campinas, SP, AprVergilio, S.R., Maldonado, J.C., Jino, M., Soares, I.W., Constraint based structural testing criteria (2006) Journal of Systems and Software, 79 (6), pp. 756-771Wong, W., Sugeta, T., Li, J.J., Maldonado, J.C., Coverage testing software architectural design in SDL (2003) Journal of Computer Networks, 42 (3), pp. 359-374. , JunEler, M.M., Delamaro, M.E., Masiero, P.C., Maldonado, J.C., Builtin structural testing of web services (2010) Proceedings of the Brazilian Symposium on Software Engineering, CBSOFT 2010, pp. 1-10Vincenzi, A.M.R., Maldonado, J.C., Wong, W.E., Delamaro, M.E., Coverage testing of Java programs and components (2005) Journal of Science of Computer ProgrammingVincenzi, A.M.R., Delamaro, M.E., Maldonado, J.C., Wong, W.E., Establishing structural testing criteria for java bytecode (2006) Softw., Pract. Exper., 36 (14), pp. 1513-1541Lemos, O.A.L., Franchin, I.G., Masiero, P.C., Integration testing of object-oriented and aspect-oriented programs: A structural pairwise approach for java (2009) Sci. Comput. Program., 74 (10), pp. 861-878Lemos, O.A.L., Vincenzi, A.M.R., Maldonado, J.C., Masiero, P.C., Control and data flow structural testing criteria for aspect-oriented programs (2007) Journal of Systems and Software, 80 (6), pp. 862-882Chaim, M.L., (1991) POKE-TOOL: A Tool to Support Data Flow Based Structural Test of Programs, , Master's thesis, DCA/FEEC/UNICAMP, Campinas, SP, Apr. (In Portuguese)Bueno, P.M.S., Jino, M., Automatic test data generation for program paths using genetic algorithms (2002) International Journal of Software Engineering and Knowledge Engineering, 12 (6), pp. 691-710Bueno, P.M., Jino, M., Wong, W.E., Diversity oriented test data generation using metaheuristic search techniques (2011) Information Sciences, , http://caocao.myipcn.org/science/article/pii/S0020025511000442, vol. In Press, Corrected Proof, pp. -, [Online]. AvailableChaim, M.L., Maldonado, J.C., Jino, M., A debugging strategy based on the requirements of testing (2004) Journal of Software Maintenance, 16 (4-5), pp. 277-308Crespo, A.N., Jino, M., Pasquini, A., Maldonado, J.C., A binomial software reliability model based on coverage of structural testing criteria (2008) Empirical Software Engineering, 13 (2), pp. 185-209. , DOI 10.1007/s10664-007-9055-3Pasquini, A., Crespo, A.N., Matrella, P., Sensitivity of reliability-growth models to operational profile errors vs testing accuracy (1996) IEEE Transactions on Reliability, 45 (4), pp. 531-540. , PII S0018952996095462Vincenzi, A.M.R., Wong, W.E., Delamaro, M.E., Maldonado, J.C., JaBUTi: A coverage analysis tool for Java programs XVII Brazilian Symposium on Software Engineering, Manaus, AM, Brasil, Oct. 2003, pp. 79-84Delamaro, M.E., Vincenzi, A.M.R., Maldonado, J.C., A strategy to perform coverage testing of mobile applications (2006) Proceedings of the 2006 International Workshop on Automation of Software Test, AST 2006, pp. 118-124Delamaro, M.E., Vincenzi, A.M.R., Structural Testing of Mobile Agents (2003) Lecture Notes on Computer Science, 2952, pp. 73-85. , International Workshop on Scientific Engineering of Java Distributed Applications (FIDJI 2003), ser. E. A. Nicolas Guelfi and G. Reggio, Eds. Springer, NovLi, J.J., Weiss, D.M., Yee, H., An automatically-generated run-time instrumenter to reduce coverage testing overhead (2008) Proceedings of the 3rd International Workshop on Automation of Software Test, AST 2008, pp. 49-56Soffa, M.L., Walcott, K.R., Mars, J., Exploiting hardware advances for software testing and debugging: Nier track (2011) Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, pp. 888-891Hutchins, M., Foster, H., Goradia, T., Ostrand, T.J., Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria (1994) ICSE, pp. 191-200Santelices, R.A., Jones, J.A., Yu, Y., Harrold, M.J., Lightweight fault-localization using multiple coverage types (2009) 31st International Conference on Software Engineering, ICSE 2009, pp. 56-66Misurda, J., Clause, J.A., Reed, J.L., Childers, B.R., Soffa, M.L., Demand-driven structural testing with dynamic instrumentation (2005) 27th International Conference on Software Engineering (ICSE 2005), pp. 156-165Santelices, R.A., Harrold, M.J., Efficiently monitoring data-flow test coverage (2007) 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2007), pp. 343-352DeMillo, R.A., Lipton, R.J., Sayward, F.G., Hints on test data selection: Help for the practicing programmer (1978) Computer, 11 (4), pp. 34-41. , AprilDeMillo, R., Guindi, D., McCracken, W., Offutt, A., King, K., An extended overview of the mothra software testing environment (1988) Software Testing, Verification, and Analysis, 1988., Proceedings of the Second Workshop on, pp. 142-151. , julAgrawal, H., DeMillo, R.A., Hathaway, R., Hsu, W., Hsu, W., Krauser, E.W., Martin, R.J., Spafford, E.H., (1989) Design of Mutant Operators for the C Programming Language, , Software Engineering Research Center, Purdue University, West Lafayette, IN, Tech. Rep. SERC-TR41-P, MarChoi, B., Mathur, A., Pattison, B., Pmothra: Scheduling mutants for execution on a hypercube (1989) SIGSOFT Softw. Eng. Notes, 14, pp. 58-65. , http://doi.acm.org/10.1145/75309.75316, November [Online]. AvailableOffutt, A.J., Pargas, R.P., Fichter, S.V., Khambekar, P.K., Mutation testing of software using a mimd computer (1992) 1992 International Conference on Parallel Processing, pp. 257-266Offutt, A.J., Rothermel, G., Zapf, C., An experimental evaluation of selective mutation (1993) ICSE '93, pp. 100-107. , http://portal.acm.org/citation.cfm?id=257572.257597, Proceedings of the 15th international conference on Software Engineering, ser. Los Alamitos, CA, USA: IEEE Computer Society Press, [Online]. AvailableOffutt, A., Pan, J., Detecting equivalent mutants and the feasible path problem (1996) Computer Assurance, 1996. COMPASS '96, 'Systems Integrity. Software Safety. Process Security'. Proceedings of the Eleventh Annual Conference on, pp. 224-236. , junOffutt, A.J., Craft, W.M., Using compiler optimization techniques to detect equivalent mutants (1994) Software Testing, Verification and Reliability, 4 (3), pp. 131-154. , http://dx.doi.org/10.1002/stvr.4370040303, [Online]. AvailableJia, Y., Harman, M., An analysis and survey of the development of mutation testing (2010) IEEE Transactions on Software Engineering, 99. , no. PrePrintsDelamaro, M.E., (1993) Proteum: Um Ambiente de Teste Baseado Na Análise de Mutantes (Proteum: A Testing Environment Based on Mutation Testing), , Master Thesis, SCE-ICMSC-USP, São Carlos - SP, OctDelamaro, M.E., Maldonado, J.C., Jino, M., Chaim, M.L., Proteum: A Testing Tool Based on Mutation Analysis Software Tools Seesion - VII Brazilian Symposium on Software Engineering, Rio de Janeiro - RJ, Oct. 1993, pp. 31-33Vincenzi, A.M.R., (1998) Subsídios para O Estabelecimento de Estratégias de Teste Baseadas Na Técnica de Mutaço Aid for Establishing Mutation Based Testing Strategies, , Master's thesis, ICMC-USP, So Carlos - SP, NovVincenzi, A.M.R., Maldonado, J.C., Barbosa, E.F., Delamaro, M.E., Unit and integration testing strategies for C programs using mutation-based criteria (2001) Software Testing, Verification and Reliability, 11 (4), pp. 249-268. , DecBarbosa, E.F., (1998) A Contribution for the Determination of A Sufficient Mutant Operators Set for C-program Testing, , Master's thesis, ICMCUSP, São Carlos, SP, Nov. (in Portuguese)Fabbri, S.C.P.F., Maldonado, J.C., Masiero, P.C., Delamaro, M.E., Mutation analysis testing for finite state machines (1994) 5th International Symposium on Software Reliability Engineering (ISSRE'94), pp. 220-229. , Monterey - CA: IEEE Computer Society Press, NovFabbri, S.C.P.F., Delamaro, M.E., Maldonado, J.C., Masieiro, P.C., Aplicaçã o da análise de mutantes na validac̃ã o de especificações baseadas em redes de petri (mutation analysis based on petri net models) VIII Brazilian Symposium on Software Engineering, Curitiba -PR, Brazil, Oct. 1994, pp. 423-437Fabbri, S.C.P.F., Maldonado, J.C., Sugeta, T., Masiero, P.C., Mutation testing applied to validate specifications based on statecharts ISSRE - International Symposium on Software Reliability Systems, Nov. 1999, pp. 210-219Simo, A.S., Maldonado, J.C., Fabbri, S.C.P.F., Proteum-RS/PN: A tool to support edition, simulation and validation of Petri nets based on mutation testing XIV Brazilian Symposium on Software Engineering, João Pessoa - PB - Brazil, Oct. 2000Delamaro, M.E., Maldonado, J.C., Mathur, A.P., Interface Mutation: An Approach for Integration Testing (2001) IEEE Transactions on Software Engineering, 27 (3), pp. 228-247. , MarAraujo, R.F., Maldonado, J.C., Delamaro, M.E., Devising Mutant Operators for Dynamic Systems Models by Applying the Hazop Study, , submitedAraujo, R.F., Maldonado, J.C., Delamaro, M.E., Tetoods Extension for Mutation Testing, , submite