Evaluating testing methods by delivered reliability

There are two main goals in testing software: (1) to achieve adequate quality (debug testing), where the objective is to probe the software for defects so that these can be removed, and (2) to assess existing quality (operational testing), where the objective is to gain confidence that the software is reliable. Debug methods tend to ignore random selection of test data from an operational profile, while for operational methods this selection is all-important. Debug methods are thought to be good at uncovering defects so that these can be repaired, but having done so they do not provide a technically defensible assessment of the reliability that results. On the other hand, operational methods provide accurate assessment, but may not be as useful for achieving reliability. This paper examines the relationship between the two testing goals, using a probabilistic analysis. We define simple models of programs and their testing, and try to answer the question of how to attain program reliability: is it better to test by probing for defects as in debug testing, or to assess reliability directly as in operational testing? Testing methods are compared in a model where program failures are detected and the software changed to eliminate them. The “better” method delivers higher reliability after all test failures have been eliminated. Special cases are exhibited in which each kind of testing is superior. An analysis of the distribution of the delivered reliability indicates that even simple models have unusual statistical properties, suggesting caution in interpreting theoretical comparisons

Predicting the Remaining Service Life of Railroad Bearings: Leveraging Machine Learning and Onboard Sensor Data

By continuously monitoring train bearing health in terms of temperature and vibration levels of bearings tested in a laboratory setting, statistical regression models have been developed to establish relationships between the sensor-acquired bearing health data with several explanatory factors that potentially influence the bearing deterioration. Despite their merits, statistical models fall short of reliable prediction accuracy levels since they entail restrictive assumptions, such as a priori known functional relationship between the response and input variables. A data-driven machine learning algorithm is presented, which can unravel the nonlinear deterioration model purely based on the bearing health data, even when the structure is not apparent. More specifically, a Gradient Boosting Machine is trained using vast amounts of laboratory data collected over the course of over a decade. This will help predict bearing failure, thus, providing railroads and railcar owners the opportunity to schedule preventive maintenance cycles rather than costly reactive ones

RESTORE: Automated Regression Testing for Datasets

In data mining, the data in various business cases (e.g., sales, marketing, and demography) gets refreshed periodically. During the refresh, the old dataset is replaced by a new one. Confirming the quality of the new dataset can be challenging because changes are inevitable. How do analysts distinguish reasonable real-world changes vs. errors related to data capture or data transformation? While some of the errors are easy to spot, the others may be more subtle. In order to detect such types of errors, an analyst will typically have to examine the data manually and assess if the data produced are "believable". Due to the scale of data, such examination is tedious and laborious. Thus, to save the analyst's time, it is important to detect these errors automatically. However, both the literature and the industry are still lacking methods to assess the difference between old and new versions of a dataset during the refresh process. In this paper, we present a comprehensive set of tests for the detection of abnormalities in a refreshed dataset, based on the information obtained from a previous vintage of the dataset. We implement these tests in automated test harness made available as an open-source package, called RESTORE, for R language. The harness accepts flat or hierarchical numeric datasets. We also present a validation case study, where we apply our test harness to hierarchical demographic datasets. The results of the study and feedback from data scientists using the package suggest that RESTORE enables fast and efficient detection of errors in the data as well as decreases the cost of testing.Comment: 10 pages, 2 figure

A Fault-Based Model of Fault Localization Techniques

Every day, ordinary people depend on software working properly. We take it for granted; from banking software, to railroad switching software, to flight control software, to software that controls medical devices such as pacemakers or even gas pumps, our lives are touched by software that we expect to work. It is well known that the main technique/activity used to ensure the quality of software is testing. Often it is the only quality assurance activity undertaken, making it that much more important. In a typical experiment studying these techniques, a researcher will intentionally seed a fault (intentionally breaking the functionality of some source code) with the hopes that the automated techniques under study will be able to identify the fault\u27s location in the source code. These faults are picked arbitrarily; there is potential for bias in the selection of the faults. Previous researchers have established an ontology for understanding or expressing this bias called fault size. This research captures the fault size ontology in the form of a probabilistic model. The results of applying this model to measure fault size suggest that many faults generated through program mutation (the systematic replacement of source code operators to create faults) are very large and easily found. Secondary measures generated in the assessment of the model suggest a new static analysis method, called testability, for predicting the likelihood that code will contain a fault in the future. While software testing researchers are not statisticians, they nonetheless make extensive use of statistics in their experiments to assess fault localization techniques. Researchers often select their statistical techniques without justification. This is a very worrisome situation because it can lead to incorrect conclusions about the significance of research. This research introduces an algorithm, MeansTest, which helps automate some aspects of the selection of appropriate statistical techniques. The results of an evaluation of MeansTest suggest that MeansTest performs well relative to its peers. This research then surveys recent work in software testing using MeansTest to evaluate the significance of researchers\u27 work. The results of the survey indicate that software testing researchers are underreporting the significance of their work

Towards quality programming in the automated testing of distributed applications

PhD ThesisSoftware testing is a very time-consuming and tedious activity and accounts for over 25% of the cost of software development. In addition to its high cost, manual testing is unpopular and often inconsistently executed. Software Testing Environments (STEs) overcome the deficiencies of manual testing through automating the test process and integrating testing tools to support a wide range of test capabilities. Most prior work on testing is in single-thread applications. This thesis is a contribution to testing of distributed applications, which has not been well explored. To address two crucial issues in testing, when to stop testing and how good the software is after testing, a statistics-based integrated test environment which is an extension of the testing concept in Quality Programming for distributed applications is presented. It provides automatic support for test execution by the Test Driver, test development by the SMAD Tree Editor and the Test Data Generator, test failure analysis by the Test Results Validator and the Test Paths Tracer, test measurement by the Quality Analyst, test management by the Test Manager and test planning by the Modeller. These tools are integrated around a public, shared data model describing the data entities and relationships which are manipulable by these tools. It enables early entry of the test process into the life cycle due to the definition of the quality planning and message-flow routings in the modelling. After well-prepared modelling and requirements specification are undertaken, the test process and the software design and implementation can proceed concurrently. A simple banking application written using Java Remote Method Invocation (RMI) and Java DataBase Connectivity (JDBC) shows the testing process of fitting it into the integrated test environment. The concept of the automated test execution through mobile agents across multiple platforms is also illustrated on this 3-tier client/server application.The National Science Council, Taiwan: The Ministry of National Defense, Taiwan

Bubble memory module

Design, fabrication and test of partially populated prototype recorder using 100 kilobit serial chips is described. Electrical interface, operating modes, and mechanical design of several module configurations are discussed. Fabrication and test of the module demonstrated the practicality of multiplexing resulting in lower power, weight, and volume. This effort resulted in the completion of a module consisting of a fully engineered printed circuit storage board populated with 5 of 8 possible cells and a wire wrapped electronics board. Interface of the module is 16 bits parallel at a maximum of 1.33 megabits per second data rate on either of two interface buses

Development of nondestructive testing techniques for honeycomb heat shields Annual report, 1 Jul. 1965 - 31 May 1966

Nondestructive testing techniques for Saturn honeycomb heat shield

Radiative Heat Transfer Analysis of Railroad Bearings for Wayside Thermal Detector Optimization

Wayside hot-box detectors (HBDs) are devices that are currently used to evaluate the health of railcar components including bearings, axles, and brakes by monitoring their temperatures. While HBDs have been instrumental in reducing some train derailments in the past few decades, the number of non-verified bearing removals has increased significantly. In general, HBDs tend to underestimate bearing temperatures in both field service and in laboratory testing, which is not surprising considering the simple two-point calibration method that is used to calibrate these devices. Because of this, different calibrations were compared and analyzed including two-point, three-point, and multi-point calibrations. Analysis of the results also suggests that the scanning location significantly affects the temperature measurement. The work in this thesis summarizes how an optimized calibration technique along with proper IR sensor alignment can markedly improve the accuracy and precision of HBD temperature measurements

Software Engineering Laboratory Series: Collected Software Engineering Papers

The Software Engineering Laboratory (SEL) is an organization sponsored by NASA/GSFC and created to investigate the effectiveness of software engineering technologies when applied to the development of application software. The activities, findings, and recommendations of the SEL are recorded in the Software Engineering Laboratory Series, a continuing series of reports that includes this document