Do System Test Cases Grow Old?

Companies increasingly use either manual or automated system testing to ensure the quality of their software products. As a system evolves and is extended with new features the test suite also typically grows as new test cases are added. To ensure software quality throughout this process the test suite is continously executed, often on a daily basis. It seems likely that newly added tests would be more likely to fail than older tests but this has not been investigated in any detail on large-scale, industrial software systems. Also it is not clear which methods should be used to conduct such an analysis. This paper proposes three main concepts that can be used to investigate aging effects in the use and failure behavior of system test cases: test case activation curves, test case hazard curves, and test case half-life. To evaluate these concepts and the type of analysis they enable we apply them on an industrial software system containing more than one million lines of code. The data sets comes from a total of 1,620 system test cases executed a total of more than half a million times over a time period of two and a half years. For the investigated system we find that system test cases stay active as they age but really do grow old; they go through an infant mortality phase with higher failure rates which then decline over time. The test case half-life is between 5 to 12 months for the two studied data sets.Comment: Updated with nicer figs without border around the

Testing in the incremental design and development of complex products

Testing is an important aspect of design and development which consumes significant time and resource in many companies. However, it has received less research attention than many other activities in product development, and especially, very few publications report empirical studies of engineering testing. Such studies are needed to establish the importance of testing and inform the development of pragmatic support methods. This paper combines insights from literature study with findings from three empirical studies of testing. The case studies concern incrementally developed complex products in the automotive domain. A description of testing practice as observed in these studies is provided, confirming that testing activities are used for multiple purposes depending on the context, and are intertwined with design from start to finish of the development process, not done after it as many models depict. Descriptive process models are developed to indicate some of the key insights, and opportunities for further research are suggested

An empirical learning-based validation procedure for simulation workflow

Simulation workflow is a top-level model for the design and control of simulation process. It connects multiple simulation components with time and interaction restrictions to form a complete simulation system. Before the construction and evaluation of the component models, the validation of upper-layer simulation workflow is of the most importance in a simulation system. However, the methods especially for validating simulation workflow is very limit. Many of the existing validation techniques are domain-dependent with cumbersome questionnaire design and expert scoring. Therefore, this paper present an empirical learning-based validation procedure to implement a semi-automated evaluation for simulation workflow. First, representative features of general simulation workflow and their relations with validation indices are proposed. The calculation process of workflow credibility based on Analytic Hierarchy Process (AHP) is then introduced. In order to make full use of the historical data and implement more efficient validation, four learning algorithms, including back propagation neural network (BPNN), extreme learning machine (ELM), evolving new-neuron (eNFN) and fast incremental gaussian mixture model (FIGMN), are introduced for constructing the empirical relation between the workflow credibility and its features. A case study on a landing-process simulation workflow is established to test the feasibility of the proposed procedure. The experimental results also provide some useful overview of the state-of-the-art learning algorithms on the credibility evaluation of simulation models

Simulation support for performance assessment of building components

The determination of performance metrics for novel building components requires that the tests are conducted in the outdoor environment. It is usually difficult to do this when the components are located in a full-scale building because of the difficulty in controlling the experiments. Test cells allow the components to be tested in realistic, but controlled, conditions. High-quality outdoor experiments and identification analysis methods can be used to determine key parameters that quantify performance. This is important for achieving standardised metrics that characterise the building component of interest, whether it is a passive solar component such as a ventilated window, or an active component such as a hybrid photovoltaic module. However, such testing and analysis does not determine how the building component will perform when placed in a real building in a particular location and climate. For this, it is necessary to model the whole building with and without the building component of interest. A procedure has been developed, and applied within several major European projects, that consists of calibrating a simulation model with high-quality data from the outdoor tests and then applying scaling and replication to one or more buildings and locations to determine performance in practice of building components. This paper sets out the methodology that has been developed and applied in these European projects. A case study is included demonstrating its application to the performance evaluation of hybrid photovoltaic modules

Degradation modeling applied to residual lifetime prediction using functional data analysis

Sensor-based degradation signals measure the accumulation of damage of an engineering system using sensor technology. Degradation signals can be used to estimate, for example, the distribution of the remaining life of partially degraded systems and/or their components. In this paper we present a nonparametric degradation modeling framework for making inference on the evolution of degradation signals that are observed sparsely or over short intervals of times. Furthermore, an empirical Bayes approach is used to update the stochastic parameters of the degradation model in real-time using training degradation signals for online monitoring of components operating in the field. The primary application of this Bayesian framework is updating the residual lifetime up to a degradation threshold of partially degraded components. We validate our degradation modeling approach using a real-world crack growth data set as well as a case study of simulated degradation signals.Comment: Published in at http://dx.doi.org/10.1214/10-AOAS448 the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org

Clinical audit project in undergraduate medical education curriculum: An assessment validation study

Objectives: To evaluate the merit of the Clinical Audit Project (CAP) in an assessment program for undergraduate medical education using a systematic assessment validation framework. Methods: A cross-sectional assessment validation study at one medical school in Western Australia, with retrospective qualitative analysis of the design, development, implementation and outcomes of the CAP, and quantitative analysis of assessment data from four cohorts of medical students (2011-2014). Results: The CAP is fit for purpose with clear external and internal alignment to expected medical graduate outcomes. Substantive validity in students’ and examiners’ response processes is ensured through relevant methodological and cognitive processes. Multiple validity features are built-in to the design, planning and implementation process of the CAP. There is evidence of high internal consistency reliability of CAP scores (Cronbach’s alpha \u3e 0.8) and inter-examiner consistency reliability (intra-class correlation\u3e0.7). Aggregation of CAP scores is psychometrically sound, with high internal consistency indicating one common underlying construct. Significant but moderate correlations between CAP scores and scores from other assessment modalities indicate validity of extrapolation and alignment between the CAP and the overall target outcomes of medical graduates. Standard setting, score equating and fair decision rules justify consequential validity of CAP scores interpretation and use. Conclusions: This study provides evidence demonstrating that the CAP is a meaningful and valid component in the assessment program. This systematic framework of validation can be adopted for all levels of assessment in medical education, from individual assessment modality, to the validation of an assessment program as a whole

Testing theory in interprofessional education: Social capital as a case study

Theory is essential to understand our interprofessional educational (IPE) practice. As a discipline, IPE has moved from being widely atheoretical to having a plethora of theories imported from the psychosocial disciplines that have utility to understand, articulate and improve IPE practice and evaluation. This paper proposes that when taking this deductive approach to theoretical development in IPE, a greater focus must now be placed on the rigorous testing of these theories within the IPE context. It synthesizes two approaches to achieving this, using the social capital theory as a case study, and focuses on the first two stages of this synthesis: first, the identification of the concepts and propositions that make up a theory within the study context and second, the value-based judgments made by the researcher and other stakeholders on the utility of these propositions. The interprofessional student group is chosen as a possible exemplar of a social network and theory-derived concepts and propositions are identified and classified within this context. With a focus on physical network characteristics, validation of these propositions with a sample of IPE educationalists is described. We present a range of propositions specifically related to the size and mix of IPE student groups, the frequency and level with which students participate in these as well as some of the existing evidence that have explored these propositions to date. Refined propositions and the way forward in the future application and empirical testing of social capital theory in IPE are presented

Combining different validation techniques for continuous software improvement - Implications in the development of TRNSYS 16

Validation using published, high quality test suites can serve to identify different problems in simulation software: modeling and coding errors, missing features, frequent sources of user confusion. This paper discusses the application of different published validation procedures during the development of a new TRNSYS version: BESTEST/ASHRAE 140 (Building envelope), HVAC BESTEST (mechanical systems) and IEA ECBCS Annex 21 / SHC Task 12 empirical validation (performance of a test cell with a very simple mechanical system). It is shown that each validation suite has allowed to identify different types of problems. Those validation tools were also used to diagnose and fix some of the identified problems, and to assess the influence of code modifications. The paper also discusses some limitations of the selected validation tools