Impact of maintainability defects on code inspections

Software inspections are effective ways to detect defects early in the development process. In this paper, we analyze the impact of certain defect types on the effectiveness of code inspection. We conducted an experiment in an academic environment with 88 subjects to empirically investigate the effect of two maintainability defects, i.e., indentation and naming conventions, on the number of functional defects found, the effectiveness of functional defect detections, and the number of false positives reported during individual code inspections. Results show that in cases where both naming conventions and indentation defects exist, the participants found minimum number of defects and reported the highest number of false positives, as compared to the cases where either indentation or naming defects exist. Among maintainability defects, indentation seems to significantly impact the number of functional defects found by the inspector, while the presence of naming conventions defects seems to have no significant impact on the number of functional defects detected. The presence of maintainability defects significantly impacts the number of false positives reported. On the effectiveness of individual code inspectors we observed no significant impact originated from the presence of indentation or naming convention defects. © 2010 ACM

Exploring the Software Verification and Validation Process with Focus on Efficient Fault Detection

Quality is an aspect of high importance in software development projects. The software organizations have to ensure that the quality of their developed products is what the customers expect. Thus, the organizations have to verify that the product is functioning as expected and validate that the product is what the customers expect. Empirical studies have shown that in many software development projects as much as half of the projected schedule is spent on the verification and validation activities. The research in this thesis focuses on exploring the state of practice of the verification and validation process and investigating methods for achieving efficient fault detection during the software development. The thesis aims at increasing the understanding of the activities conducted to verify and validate the software products, by the means of empirical research in the software engineering domain. A survey of eleven Swedish software development organizations investigates the current state of practice of the verification and validation activities, and how these activities are managed today. The need for communicating and visualising the verification and validation process was expressed during the survey. Therefore the usefulness of process simulations was evaluated in the thesis. The simulations increased the understanding of the relationships between different activities among the involved participants. In addition, an experiment was conducted to compare the performance of the two verification and validation activities, inspection and testing. In the future work, empirical research, including experiment results, will be used for calibration and validation of simulation models, with focus on using simulation as a method for decision support in the verification and validation process

Testing og feilsøking av program : Testteknikkar for feilsøking i eit SAS-program

Masteroppgåva prøver å besvara problemstillingar om val av testteknikkar for testing og feilsøking generelt, samt korleis Riksarkivet sitt SAS-program Arkade best bør testast/feilsøkast. Testteknikkar har innverknad på kva for feil som blir funne, og i praksis opplever mange testarar at ulike testteknikkar er komplementære til kvarandre (dei finn ulike feil). Men det er også mange andre forhold som har innverknad, og ofte er det uråd å fastslå forklaringsvariablane sin relative innverknad. Det er mange forhold som er felles for testing og debugging for dei generelle og spesielle problemstillingane. Det indikerer at desse forholda er allmengyldige. Eksempler på slike forhold er utføringen av ein testteknikk, testverktøy, den enkelte feil sin natur, det tekniske testmiljøet, innstillinga til å finna feil, testkompetanse, utviklingsmetodikk og programmerngsstil. Eg har valt testteknikken utforskande testing som utgangspunkt for å testa Arkade direkte eller til å finna andre testteknikkar som passar til ulike situasjonar. Oppgåva diskuterer fordeler og ulemper med denne testteknikken. Eg brukar testteknikken til å utvikla ein testteknikk for å analysera SAS-loggen. Hensikten er å kvalitetssikra og effektivisera gjennomgangen av SAS-loggen etter kodeeksekvering. Eg brukar også utforskande testing til design av testtilfelle for dynamisk testing av Arkade. Rapporten konkluderer med at Arkade kan bli lettare testbar ved å forbetra programmeringsstilen. Eit anna forslag er bruk av globale makrovariablar til erstatning for bortkommentering av testkode, slik at aktivering og deaktivering av bortkommentering av testkode i kjeldekoden enkelt kan gjerast ved berre å endra verdien på ein makrovariabel. Rapporten foreslå også at Arkade-applikasjonen bør få implementert større robustheit for feilhåndtering. Kvar makro bør få si eiga programfil, slik at makroane blir lettare å finna når ein treng dei. Applikasjonen kan også gjerast meir testbar ved å aktivera ei rekke systemopsjonar i SAS som gjer at innebygd funksjonalitet for debugging blir brukt. Dynamisk testing av Arkade ved eksekvering av 23 testtilfelle har avdekka 9 feil. 14 testtilfelle køyrde feilfritt. 1 feil er ikkje funne årsaken til. 8 av feila er kategorisert som dårleg feilhåndtering (2 feil), feil i ein spesifikk del av programmet (3 feil) og for dårleg validering (3 feil) av samsvar mellom inndata-filer (datafiler og metadatafil)

Análisis de la adopción y uso de las revisiones de software

Incluye bibliografía.El trabajo aborda un tema de indudable relevancia científica, tanto teórica como práctica: el análisis de la adopción y el uso de las revisiones de software

The Effectiveness of <i>t</i>-Way Test Data Generation

Modern society is increasingly dependent on the correct functioning of software and increasingly so in areas that are considered safety related or safety critical. Therefore, there is an increasing need to be able to verify and validate that the software is in fact correct and will perform its intended function. Many approaches to this problem have been proposed; however, none seems likely to supplant the role of testing in the near future. If we accept that there is, and will be, a continuing need to be able to test software then the question becomes one of how can this be done effectively, both in terms of ability to detect errors and in terms of cost. One avenue of research that offers prospects of improving both of these aspects is the automatic generation of test data. There has recently been a large amount of work conducted in this area. One particularly promising direction has been the application of ideas from the field of experimental design and in particular, the field of t-way adequate factorial designs. The area however, is not without issues; there is evidence that the technique is capable of detecting errors but that evidence is not unequivocal. Moreover, as with almost all work in the area of automatic test generation, there has been very little comparative work comparing the technique with other test data generation techniques. Worse, there has been effectively no work done that compares any automatic test data generation technique with the effectiveness of tests generated by humans. Another major issue with the technique is the number of tests that applying the technique can result in. This implies that there is a need for an automated oracle if the technique is to be successfully applied. The flaw with this is of course that in most situations the oracle is the human that is conducting the tests, a point often ignored in testing research. The work presented here addresses both of these points. To do this I have used a code base taken from an industrial engine control system that has an existing set of high quality unit tests developed by hand. To complement this, several other techniques for automatically generating test data have been applied, namely random testing, random experimental designs and a technique for generating single factor experiments. To address the issue of being able to compare the error detection ability of all of the sets of test vectors, rather than the usual effectiveness surrogates of code coverage I have used mutation analysis on the code base to directly measure the ability of each set of test vectors to discover common coding errors. The results presented here show that test data generation techniques based on t-way factorial designs are at least as effective as handgenerated tests and superior to random testing and the factor experimental technique. The oracle problem associated with the factorial design techniques was addressed using a test set minimisation approach. The mutation tool monitored which vectors could “kill” which code mutants. After a subset of the test vectors had been run, the most effective vectors were retained and the rest discarded. Likewise, mutants that were killed were removed from further consideration and the process repeated. Experimental results show that this minimisation procedure is effective at reducing computational overhead and is capable of producing final sets of test vectors that are comparable in size with the sets of hand-generated tests and so amenable to final hand checking

Studying the Effects of Code Inspection and Structural Testing on Software Quality

The most common techniques for detecting defects in software artifacts are inspection and testing. Since both techniques are effort consuming, they are often presented as being counterparts or even rivals rather than as being complementary. Hence, few controlled empirical studies investigate the effects of inspection and testing on software quality when applied in sequence. This paper contributes a controlled experiment to shed light on this issue. Twenty subjects performed sequentially code inspection and structural testing using different coverage values as test criteria on a C-code module. We adopted this sequence because it is recommended for use in industry. The results of this experiment show that inspection significantly outperforms structural testing with respect to (cost-)effectiveness for defect detection. Furthermore, the experimental results indicate little evidence to support the hypothesis that structural testing detects defects of a particular class that were missed b y inspection and vice versa. These findings lead us to the conclusion that inspection and structural testing do not complement each other well. In fact, prior inspection seems to hinder the (cost-)effectiveness of structural testing. Since inspection out-performs structural testing and since 39 percent (on average) of the defects were not detected at all, it might be more valuable to apply inspection together with other testing techniques, such as boundary value analysis, to achieve a better defect coverage. We are aware that a single experiment does not provide conclusive evidence. Hence, we consider it only one step in the determination of the optimal mix of defect detection techniques. Additional research as well as replication of this experiment are required to make further progress into this direction