Industrial Surveys on Software Testing Practices: A Literature Review

A US government agency estimated the national cost of inadequate software testing to be \$60 billion annually, and that was 20 years ago. As the role of technology and software has been rapidly increasing worldwide for decades, it suffices to say that the worldwide fiscal effect of poor testing practices today is probably ``quite a bit``. An increasing number of industry-focused survey studies on testing have been published worldwide in recent years, signalling an increased need to characterize the testing practices of the software development industry. These types of studies can help to guide future research efforts towards subjects that are meaningful to the industry, and provide practitioners with an opportunity to compare their own practice to those of their peers and recognize the main improvement areas. As no secondary study devoted to these types of survey studies could be identified, the opportunity was seized to carry out a literature review was to find out what the data from these studies can tell us when aggregated. The precise topics focused on were the usage of test levels, test types, test design techniques, test tools and test automation. Looking at these studies in aggregate tells us about some general trends: unit testing, functional testing and regression testing are popular everywhere, and also quite popular regardless of the surveyed population are performance testing and usability testing. The popularity of the other test levels and test types vary from survey to survey or region to region. Black-box techniques and experience-based techniques are more popular than white-box techniques. Exploratory testing, error guessing, use case testing and boundary value analysis are some of the most popular test design techniques. Much of the industry relies on manual testing over automated testing and/or have inadequately adopted the usage of testing tools

Software testing education and training in Hong Kong

While the use of computer applications is widely spread in every business and, hence, the reliability of software is critical, it is believed that many organizations involved in software development do not take software testing sufficiently seriously as an important task. It is worthwhile to find out how far organizations are carrying out software testing in a systematic and structured manner or still taking on an ad-hoc approach. A survey was conducted to understand the software testing practices and the level of related education and training in Hong Kong. It was found that most testing team members did not have formal training in software testing. University curricula generally did not prepare graduates with enough coverage in software testing. It is proposed that a review of the current software engineering curricula in the universities to examine the coverage of software testing will be useful to the development of quality software. © 2005 IEEE.published_or_final_versio

Experiences in Software Testing Education: Some Observations from an International Cooperation

This paper examines the outcomes of teaching a course in Software Testing in Ireland and China over a two-year period. In both institutions the delivery of the course is constrained to two-week duration. The learning objectives for this course are explained. The outcomes of the courses in terms of student learning are compared and analyzed. A number of observations are made that lead to recommendations as to how the educational experience can be improved for the students in both countries

Test adequacy assessment using test-defect coverage analytic model

Software testing is an essential activity in software development process that has been widely used as a means of achieving software reliability and quality. The emergence of incremental development in its various forms required a different approach to determining the readiness of the software for release. This approach needs to determine how reliable the software is likely to be based on planned tests, not defect growth and decline as typically shown in reliability growth models. A combination of information from a number of sources into an easily understood dashboard is expected to provide both qualitative and quantitative analyses of test and defect coverage properties. Hence, Test-Defect Coverage Analytic Model (TDCAM) is proposed which combines test and defect coverage information presented in a dashboard to help deciding whether there are enough tests planned. A case study has been conducted to demonstrate the usage of the proposed model. The visual representations and results gained from the case study show the benefits of TDCAM in assisting practitioners making informed test adequacy-related decisions

Towards a hybrid testing process unifying exploratory testing and scripted testing

CONTEXT Given the current state of the art in research, practitioners are faced with the challenge of choosing scripted testing (ST) or exploratory testing (ET). OBJECTIVE This study aims at systematically incorporating strengths of ET and ST in a hybrid testing process to overcome the weaknesses of each. METHOD We utilized systematic review and practitioner interviews to identify strengths and weaknesses of ET and ST. Strengths of ET were mapped to weaknesses of ST and vice versa. Noblit and Hare's lines-ofargument method was used for data analysis. The results of the mapping were used as input to codesign a hybrid process with experienced practitioners. RESULTS We found a clear need to create a hybrid process as follows: (i) both ST and ET provide strengths and weaknesses, and these depend on some particular conditions, which prevents preference of one approach to another; and (ii) the mapping showed that it is possible to address the weaknesses in one process by the strengths of the other in a hybrid form. With the input from literature and industry experts, a flexible and iterative hybrid process was designed. CONCLUSIONS Practitioners can clearly benefit from using a hybrid process given the mapping of advantages and disadvantage

Goal-Oriented Mutation Testing with Focal Methods

Mutation testing is the state-of-the-art technique for assessing the fault-detection capacity of a test suite. Unfortunately, mutation testing consumes enormous computing resources because it runs the whole test suite for each and every injected mutant. In this paper we explore fine-grained traceability links at method level (named focal methods), to reduce the execution time of mutation testing and to verify the quality of the test cases for each individual method, instead of the usually verified overall test suite quality. Validation of our approach on the open source Apache Ant project shows a speed-up of 573.5x for the mutants located in focal methods with a quality score of 80%.Comment: A-TEST 201

An Empirical Investigation of Software Testing Methods and Techniques in the Province of Vojvodina

A high-quality test design is a conditio sine qua non of successful software testing process, and its effectiveness depends, among other things, on the choice and proper use of appropriate methods and relevant software testing techniques. The main goal of this study was to provide insight into the use of current methods and relevant software testing techniques used in the test design phase of software testing process in software companies in the Province of Vojvodina. The empirical study was conducted by a survey research strategy in twenty-four software organisations. Eighty-three respondents took part in the survey. Descriptive analysis, correlation analysis, hierarchical cluster analysis, the multidimensional scaling, binomial test and Cohran\u27s Q test were used for analyzing gathered quantitative data. The survey results have shown that respondents use to a significant extent the techniques belonging to ISO/IEC/IEEE 29119 testing standard. Comparison of the gathered data with individual results of similar studies conducted in Canada, Australia and Turkey has shown similarities between them and companies in the Province of Vojvodina. The findings of this study present empirically verified recommendations for testing design phase realization in the form of least and most used software testing methods and techniques, their benefits, limitations and details in application, similarities between software testing techniques, software testing techniques clusters and the probability of use of individual techniques

A Content Analysis-Based Approach to Explore Simulation Verification and Identify Its Current Challenges

Verification is a crucial process to facilitate the identification and removal of errors within simulations. This study explores semantic changes to the concept of simulation verification over the past six decades using a data-supported, automated content analysis approach. We collect and utilize a corpus of 4,047 peer-reviewed Modeling and Simulation (M&S) publications dealing with a wide range of studies of simulation verification from 1963 to 2015. We group the selected papers by decade of publication to provide insights and explore the corpus from four perspectives: (i) the positioning of prominent concepts across the corpus as a whole; (ii) a comparison of the prominence of verification, validation, and Verification and Validation (V&V) as separate concepts; (iii) the positioning of the concepts specifically associated with verification; and (iv) an evaluation of verification\u27s defining characteristics within each decade. Our analysis reveals unique characterizations of verification in each decade. The insights gathered helped to identify and discuss three categories of verification challenges as avenues of future research, awareness, and understanding for researchers, students, and practitioners. These categories include conveying confidence and maintaining ease of use; techniques\u27 coverage abilities for handling increasing simulation complexities; and new ways to provide error feedback to model users