Robust Dynamic Selection of Tested Modules in Software Testing for Maximizing Delivered Reliability

Software testing is aimed to improve the delivered reliability of the users. Delivered reliability is the reliability of using the software after it is delivered to the users. Usually the software consists of many modules. Thus, the delivered reliability is dependent on the operational profile which specifies how the users will use these modules as well as the defect number remaining in each module. Therefore, a good testing policy should take the operational profile into account and dynamically select tested modules according to the current state of the software during the testing process. This paper discusses how to dynamically select tested modules in order to maximize delivered reliability by formulating the selection problem as a dynamic programming problem. As the testing process is performed only once, risk must be considered during the testing process, which is described by the tester's utility function in this paper. Besides, since usually the tester has no accurate estimate of the operational profile, by employing robust optimization technique, we analysis the selection problem in the worst case, given the uncertainty set of operational profile. By numerical examples, we show the necessity of maximizing delivered reliability directly and using robust optimization technique when the tester has no clear idea of the operational profile. Moreover, it is shown that the risk averse behavior of the tester has a major influence on the delivered reliability.Comment: 19 pages, 4 figure

Evaluating testing methods by delivered reliability

There are two main goals in testing software: (1) to achieve adequate quality (debug testing), where the objective is to probe the software for defects so that these can be removed, and (2) to assess existing quality (operational testing), where the objective is to gain confidence that the software is reliable. Debug methods tend to ignore random selection of test data from an operational profile, while for operational methods this selection is all-important. Debug methods are thought to be good at uncovering defects so that these can be repaired, but having done so they do not provide a technically defensible assessment of the reliability that results. On the other hand, operational methods provide accurate assessment, but may not be as useful for achieving reliability. This paper examines the relationship between the two testing goals, using a probabilistic analysis. We define simple models of programs and their testing, and try to answer the question of how to attain program reliability: is it better to test by probing for defects as in debug testing, or to assess reliability directly as in operational testing? Testing methods are compared in a model where program failures are detected and the software changed to eliminate them. The “better” method delivers higher reliability after all test failures have been eliminated. Special cases are exhibited in which each kind of testing is superior. An analysis of the distribution of the delivered reliability indicates that even simple models have unusual statistical properties, suggesting caution in interpreting theoretical comparisons

Application of a failure driven test profile in random testing

Random testing techniques have been extensively used in reliability assessment, as well as in debug testing. When used to assess software reliability, random testing selects test cases based on an operational profile; while in the context of debug testing, random testing often uses a uniform distribution. However, generally neither an operational profile nor a uniform distribution is chosen from the perspective of maximizing the effectiveness of failure detection. Adaptive random testing has been proposed to enhance the failure detection capability of random testing by evenly spreading test cases over the whole input domain. In this paper, we propose a new test profile, which is different from both the uniform distribution, and operational profiles. The aim of the new test profile is to maximize the effectiveness of failure detection. We integrate this new test profile with some existing adaptive random testing algorithms, and develop a family of new random testing algorithms. These new algorithms not only distribute test cases more evenly, but also have better failure detection capabilities than the corresponding original adaptive random testing algorithms. As a consequence, they perform better than the pure random testing

Deriving a frequentist conservative confidence bound for probability of failure per demand for systems with different operational and test profiles

Reliability testing is typically used in demand-based systems (such as protection systems) to derive a confidence bound for a specific operational profile. To be realistic, the number of tests for each class of demand should be proportional to the demand frequency of the class. In practice however, the actual operational profile may differ from that used during testing. This paper provides a means for estimating the confidence bound when the test profile differs from the profile used in actual operation. Based on this analysis the paper examines what bound can be claimed for different types of profile uncertainty and options for dealing with this uncertainty. We also show that the same conservative bound estimation equations can be applied to cases where different measures of software test coverage and operational profile are used

Improved Software Testing for Open Architecture

Proceedings Paper (for Acquisition Research Program)Applying traditional manual US Navy testing practices to OA systems will limit many benefits of OA, such as system scalability, rapid configuration changes, and effective component reuse. Pairing profile-driven automated software testing with test reduction techniques should enable these benefits and keep resource requirements at feasible levels. Test cases generated by operational profiles have been shown to be more effective than those developed by other methods, such as random or selective testing, and more resource-efficient than exhaustive approaches. This research effort increases the fidelity of the operational profile, creating an environment model referred to as a High-Fidelity Profile Model (HFPM) that can statistically describe individual software inputs. Samples from the HFPM''s probability distributions can generate operationally realistic or overly-stressful test cases for software modules under test. This process can be automated and paired with output checking functions, enabling automated effective software testing, and potentially improving reliability. Such models would be ideal for US Navy Open Architecture (OA) software because of the defined interface standards. HFPMs can enable effective testing in software reuse applications and are ideal for testing multiple releases of maturing software. This research defines the HFPM, presents a methodology to develop, validate, and apply it.Naval Postgraduate School Acquisition Research ProgramApproved for public release; distribution is unlimited

Tool support for statistical testing of software components

We describe the "STSC" prototype tool that supports the statistical testing of software components. The tool supports a wide range of operational profiles and test oracles for test case generation and output evaluation. The tool also generates appropriate values for different types of input parameters of operations. STSC automatically generates a test driver from an operational profile. This test driver invokes a test oracle that is implemented as a behaviour-checking version of the implementation. To evaluate the flexibility and usability of the tool, it has been applied to several case studies using different types of operational profiles and test oracles

Humidity Testing for Human Rated Spacecraft

Determination that equipment can operate in and survive exposure to the humidity environments unique to human rated spacecraft presents widely varying challenges. Equipment may need to operate in habitable volumes where the atmosphere contains perspiration, exhalation, and residual moisture. Equipment located outside the pressurized volumes may be exposed to repetitive diurnal cycles that may result in moisture absorption and/or condensation. Equipment may be thermally affected by conduction to coldplate or structure, by forced or ambient air convection (hot/cold or wet/dry), or by radiation to space through windows or hatches. The equipment s on/off state also contributes to the equipment s susceptibility to humidity. Like-equipment is sometimes used in more than one location and under varying operational modes. Due to these challenges, developing a test scenario that bounds all physical, environmental and operational modes for both pressurized and unpressurized volumes requires an integrated assessment to determine the "worst-case combined conditions." Such an assessment was performed for the Constellation program, considering all of the aforementioned variables; and a test profile was developed based on approximately 300 variable combinations. The test profile has been vetted by several subject matter experts and partially validated by testing. Final testing to determine the efficacy of the test profile on actual space hardware is in the planning stages. When validation is completed, the test profile will be formally incorporated into NASA document CxP 30036, "Constellation Environmental Qualification and Acceptance Testing Requirements (CEQATR).

On the need for bump event correction in vibration test profiles representing road excitations in automobiles

This paper presents an approach to the synthesis of compressed vibration test profiles representing much longer time histories obtained in road testing of ground vehicles. Vibration test profiles are defined as those related directly to operational testing on specific road surfaces and which summarise the input to the vehicle in the given conditions. The method extends classical Fourier transform technique by means of bump event correction in the background Fourier signal where the bump event term implies a high-amplitude transient event of the shock type. The orthogonal wavelet decomposition was used as a specific filtering tool facilitating bump event identification. Examples of seat guide vertical acceleration have been considered. Calculated probability density functions suggest the ability of the bump correction method to improve the statistical accuracy of the final vibration test profile with respect to the original road data. Test profiles obtained by means of Fourier transform synthesis with subsequent reinsertion of bump events from separated frequency bands were more accurate than those obtained by Fourier synthesis alone. Further developments led to advanced bump reinsertion with synchronisation of events occurring in different frequency bands at the same moment of time. Test profiles generated in this way have provided better accuracy compared to the non-synchronised algorithm

Developments on a Cold Bead Pull Test Stand for SRF Cavities

Final tuning and field profile characterization of SRF cavities always takes place at room temperature. However, important questions remains as to what happens when the cavity is cooled to LHe temperature, in particular with multi cell systems. To enable the characterization of cavities in the cold, we have designed and commissioned a cold bead pull test stand at HZB. The present test stand is designed to be integrated in HoBiCaT Horizontal bi cavity testing facility [1] with the ability to provide electric field profile measurements under realistic superconducting conditions T 1.8K . In this paper mechanical and operational details of the apparatus will be described as well as future plans for the development and usage of this facilit