Comparing the effectiveness of equivalence partitioning, branch testing and code reading by stepwise abstraction applied by subjects

Some verification and validation techniques have been evaluated both theoretically and empirically. Most empirical studies have been conducted without subjects, passing over any effect testers have when they apply the techniques. We have run an experiment with students to evaluate the effectiveness of three verification and validation techniques (equivalence partitioning, branch testing and code reading by stepwise abstraction). We have studied how well able the techniques are to reveal defects in three programs. We have replicated the experiment eight times at different sites. Our results show that equivalence partitioning and branch testing are equally effective and better than code reading by stepwise abstraction. The effectiveness of code reading by stepwise abstraction varies significantly from program to program. Finally, we have identified project contextual variables that should be considered when applying any verification and validation technique or to choose one particular technique

Checklist based reading's influence on a developer's understanding

This paper addresses the influence the Checklist Based Reading inspection technique has on a developer's ability to modify inspected code. Traditionally, inspections have been used to detect defects within the development life cycle. This research identified a correlation between the number of defects detected and the successful code extensions for new functionality unrelated to the defects. Participants reported that having completed a checklist inspection, modifying the code was easier because the inspection had given them an understanding of the code that would not have existed otherwise. The results also showed a significant difference in how developers systematically modified code after completing a checklist inspection when compared to those who had not performed a checklist inspection. This study has shown that applying software inspections for purposesother than defect detection include software understanding and comprehension

Proceedings of the 14th Annual Software Engineering Workshop

Several software related topics are presented. Topics covered include studies and experiment at the Software Engineering Laboratory at the Goddard Space Flight Center, predicting project success from the Software Project Management Process, software environments, testing in a reuse environment, domain directed reuse, and classification tree analysis using the Amadeus measurement and empirical analysis

Optimizing Usability Studies by Complementary Evaluation Methods

This paper examines combinations of complementary evaluation methods as a strategy for efficient usability problem discovery. A data set from an earlier study is re-analyzed, involving three evaluation methods applied to two virtual environment applications. Results of a mixed-effects logistic regression suggest that usability testing and inspection discover rather disjunctive sets of problems. A resampling analysis reveals that mixing inspection and usability testing sessions in equal parts finds 20% more problems with the same number of session

The cleanroom case study in the Software Engineering Laboratory: Project description and early analysis

This case study analyzes the application of the cleanroom software development methodology to the development of production software at the NASA/Goddard Space Flight Center. The cleanroom methodology emphasizes human discipline in program verification to produce reliable software products that are right the first time. Preliminary analysis of the cleanroom case study shows that the method can be applied successfully in the FDD environment and may increase staff productivity and product quality. Compared to typical Software Engineering Laboratory (SEL) activities, there is evidence of lower failure rates, a more complete and consistent set of inline code documentation, a different distribution of phase effort activity, and a different growth profile in terms of lines of code developed. The major goals of the study were to: (1) assess the process used in the SEL cleanroom model with respect to team structure, team activities, and effort distribution; (2) analyze the products of the SEL cleanroom model and determine the impact on measures of interest, including reliability, productivity, overall life-cycle cost, and software quality; and (3) analyze the residual products in the application of the SEL cleanroom model, such as fault distribution, error characteristics, system growth, and computer usage

Temporal behavior of defect detection performance in design documents

Die Qualität der Software ist natürlich ein erfolgskritischer Faktor im Software Engineering (SE), genauso wie die Design Dokumente in den frühen Softwareentwicklungsphasen. Organisatorische Faktoren, wie etwa der verwendete Software-Entwicklungsprozess, helfen den Prozeß an sich besser zu Strukturieren und zu Optimieren. Entwicklungsansätze unterstützen diesen Prozeß, während analytische Ansätze darauf abzielen Fehler und Produktabweichungen zu vermeiden. Software Inspektionen (SI) und Tests sind bereits bekannte und anerkannte Techniken im SE um Fehler im Software Code, in Spezifikationen oder Design Dokumenten, während verschiedenster Phasen des Produktlebenszykluses, zu identifizieren. Ein Hauptaugenmerk von analytischen Qualitätssicherungen wie SI und Tests liegt auf der frühen Entdeckung von Fehlern. Denn je später ein Fehler im Produktentwicklungsprozess gefunden wird, desto aufwendiger und teurer ist dessen Entfernung. SI fokussieren auf eine Fehlerfindung in einer sehr frühen Phase des gesamten Prozesses ohne die Notwendigkeit eines Ausführbaren Software Codes. Deshalb ist SI anwendbar auf geschriebene Text Doku-ment wie Design Dokumente. Traditionelle Testansätze fokussieren auf die Erstellung von Testfällen und deren Exekution in späteren Phasen des Prozesses, weil sie im Gegensatz zu SI auf ausführbaren Code angewiesen sind. Folgernd ist es notwendig Testfallerstellung und SI zu kombinieren, um in noch frühen Phasen die Qualität weiter verbessern zu können. Die Vorteile beider Ansätze zu vereinen wird helfen um (a) Fehler sehr früh zu finden und (b) Testfälle zu definieren, welche ein systematisches Testen erlauben, daß wiederum auf Anfor-derungen und Use-Cases basiert. Der Ansatz in dieser These - auf Inspektionen basiertes Tes-ten – wird zu einer „Zuerst Testen“ Strategie auf Anforderungsbasis führen Diese These konzentriert sich auf einen auf Inspektionen basierten Test Ansatz, sowie auf SI generell mit einer genaueren Untersuchung des zeitlichen Verhaltens dieser Techniken in Design Dokumenten mit Hauptaugenmerk auf sehr kritische und kritische Fehler. Die Ergebnisse der Untersuchungen des zeitlichen Verhaltens ergaben, daß UBR in dem Zeit-intervall der ersten 120 Minuten äußerst effektiv und effizient agiert. UBT-i hingegen benötigt mehr Zeit, ca. 44 % um ein gleichwertiges Ergebnis erzielen zu können. Der Vergleich der bei-den Software Fehlerfindungstechniken zeigte weiters, daß UBR ganzheitlich gesehen nicht die überlegene Technik ist. Wegen der inkonsistenten Resultate der Experiment Sessions kann jedoch auch keine überlegene Technik definitiv genannt werden. Betreffend den Ergebnissen der False Positives, konnte das erwartete zeitliche Verhalten, daß die wenigsten False Positi-ves in den ersten 120 Minuten gefunden werden, nicht beobachtet werden. Deshalb mußte die betreffende Hypothese verworfen werden. Die These basiert auf einem Experiment, welches in einer kontrollierten akademischen Umge-bung durchgeführt wurde um die Fehlerfindungseffizienz Einzelner zu untersuchen. Die Ergebnisse werden Projekt- und Qualitätsmanagern helfen, um deren Qualitätsmaßnahmen besser planen zu können und es weiters ermöglichen deren zeitliche Dauer und daraus folgende Effizienz und Effektivität besser abschätzen zu können.The quality of software requirements and design documents are success critical issues in soft-ware engineering (SE) practice. Organizational measures, e.g., software processes, help struc-turing the development process along the project life-cycle, constructive approaches support building software products, and analytical approaches aim at investigating deliverables with respect to defects and product deviations. Software inspection and testing are well-known and common techniques in Software Engineering to identify defects in code documents, specifica-tions, and requirements documents in various phases of the project life-cycle. A major goal of analytical quality assurance activities, e.g., inspection and testing, is the detection of defects as early as possible because rework effort and cost increase, if defects are identified late in the project. Software inspection (SI) focuses on defect detection in early phases of software development without the need for executable software code. Thus, SI is applicable to written text documents, e.g., specification and requirements documents. Traditional testing approaches focus on test case definition and execution in later phases of development because testing requires executable code. Thus, we see the need to combine test case generation and software inspection early in the software project to increase software product quality and test cases. Bundling benefits from early defect detection (SI application) and early test case definition based on SI results can help identifying (a) defects early and (b) derive test cases definitions for systematic testing based on requirements and use cases. Our approach – inspection-based testing – leads to a test-first strategy on requirements level. This thesis focuses on the investigation of an inspection-based testing approach and software inspection with respect to the temporal behavior of defect detection with emphasis on critical defects in requirements and specification documents. The outcomes concerning the temporal behavior showed up some interesting results. UBR performs in the time interval of the first 120 minutes very effective and efficient. UBT-i in con-trary needs more time, about 44 % for its testing duration to achieve as good defect detection results as UBR. The comparison of these two software fault detection techniques showed that UBR is on the whole not the superior technique. Because of the inconsistent findings in the experiment sessions a clear favorite cannot be named. Concerning the results for the fault positives the expected temporal behavior, which was that the fewest false positives were found in the first 120 minutes, could not be investigated and the hypothesis on this had to be rejected. A controlled experiment in an academic environment was made to investigate defect detection performance and the temporal behavior of defect detection for individuals in a business IT software solution. The results can help project and quality managers to better plan analytical quality assurance activities, i.e., inspection and test case generation, with respect to the temporal behavior of both defect detection approaches

Software Engineering Laboratory Series: Proceedings of the Twentieth Annual Software Engineering Workshop

The Software Engineering Laboratory (SEL) is an organization sponsored by NASA/GSFC and created to investigate the effectiveness of software engineering technologies when applied to the development of application software. The activities, findings, and recommendations of the SEL are recorded in the Software Engineering Laboratory Series, a continuing series of reports that includes this document

Proceedings of the Twenty-Third Annual Software Engineering Workshop

The Twenty-third Annual Software Engineering Workshop (SEW) provided 20 presentations designed to further the goals of the Software Engineering Laboratory (SEL) of the NASA-GSFC. The presentations were selected on their creativity. The sessions which were held on 2-3 of December 1998, centered on the SEL, Experimentation, Inspections, Fault Prediction, Verification and Validation, and Embedded Systems and Safety-Critical Systems