Quality-Aware Learning to Prioritize Test Cases

Software applications evolve at a rapid rate because of continuous functionality extensions, changes in requirements, optimization of code, and fixes of faults. Moreover, modern software is often composed of components engineered with different programming languages by different internal or external teams. During this evolution, it is crucial to continuously detect unintentionally injected faults and continuously release new features. Software testing aims at reducing this risk by running a certain suite of test cases regularly or at each change of the source code. However, the large number of test cases makes it infeasible to run all test cases. Automated test case prioritization and selection techniques have been studied in order to reduce the cost and improve the efficiency of testing tasks. However, the current state-of-art techniques remain limited in some aspects. First, the existing test prioritization and selection techniques often assume that faults are equally distributed across the software components, which can lead to spending most of the testing budget on components less likely to fail rather than the ones highly to contain faults. Second, the existing techniques share a scalability problem not only in terms of the size of the selected test suite but also in terms of the round-trip time between code commits and engineer feedback on test cases failures in the context of Continuous Integration (CI) development environments. Finally, it is hard to algorithmically capture the domain knowledge of the human testers which is crucial in testing and release cycles. This thesis is a new take on the old problem of reducing the cost of software testing in these regards by presenting a data-driven lightweight approach for test case prioritization and execution scheduling that is being used (i) during CI cycles for quick and resource-optimal feedback to engineers, and (ii) during release planning by capturing the testers domain knowledge and release requirements. Our approach combines software quality metrics with code churn metrics to build a regressive model that predicts the fault density of each component and a classification model to discriminate faulty from non-faulty components. Both models are used to guide the testing effort to the components likely to contain the largest number of faults. The predictive models have been validated on eight industrial automotive software applications at Daimler, showing a classification accuracy of 89% and an accuracy of 85.7% for the regression model. The thesis develops a test cases prioritization model based on features of the code change, the tests execution history and the component development history. The model reduces the cost of CI by predicting whether a particular code change should trigger the individual test suites and their corresponding test cases. In order to algorithmically capture the domain knowledge and the preferences of the tester, our approach developed a test case execution scheduling model that consumes the testers preferences in the form of a probabilistic graph and solves the optimal test budget allocation problem both online in the context of CI cycles and offline when planning a release. Finally, the thesis presents a theoretical cost model that describes when our prioritization and scheduling approach is worthwhile. The overall approach is validated on two industrial analytical applications in the area of energy management and predictive maintenance, showing that over 95% of the test failures are still reported back to the engineers while only 43% of the total available test cases are being executed

Quality-Aware Learning to Prioritize Test Cases

Software applications evolve at a rapid rate because of continuous functionality extensions, changes in requirements, optimization of code, and fixes of faults. Moreover, modern software is often composed of components engineered with different programming languages by different internal or external teams. During this evolution, it is crucial to continuously detect unintentionally injected faults and continuously release new features. Software testing aims at reducing this risk by running a certain suite of test cases regularly or at each change of the source code. However, the large number of test cases makes it infeasible to run all test cases. Automated test case prioritization and selection techniques have been studied in order to reduce the cost and improve the efficiency of testing tasks. However, the current state-of-art techniques remain limited in some aspects. First, the existing test prioritization and selection techniques often assume that faults are equally distributed across the software components, which can lead to spending most of the testing budget on components less likely to fail rather than the ones highly to contain faults. Second, the existing techniques share a scalability problem not only in terms of the size of the selected test suite but also in terms of the round-trip time between code commits and engineer feedback on test cases failures in the context of Continuous Integration (CI) development environments. Finally, it is hard to algorithmically capture the domain knowledge of the human testers which is crucial in testing and release cycles. This thesis is a new take on the old problem of reducing the cost of software testing in these regards by presenting a data-driven lightweight approach for test case prioritization and execution scheduling that is being used (i) during CI cycles for quick and resource-optimal feedback to engineers, and (ii) during release planning by capturing the testers domain knowledge and release requirements. Our approach combines software quality metrics with code churn metrics to build a regressive model that predicts the fault density of each component and a classification model to discriminate faulty from non-faulty components. Both models are used to guide the testing effort to the components likely to contain the largest number of faults. The predictive models have been validated on eight industrial automotive software applications at Daimler, showing a classification accuracy of 89% and an accuracy of 85.7% for the regression model. The thesis develops a test cases prioritization model based on features of the code change, the tests execution history and the component development history. The model reduces the cost of CI by predicting whether a particular code change should trigger the individual test suites and their corresponding test cases. In order to algorithmically capture the domain knowledge and the preferences of the tester, our approach developed a test case execution scheduling model that consumes the testers preferences in the form of a probabilistic graph and solves the optimal test budget allocation problem both online in the context of CI cycles and offline when planning a release. Finally, the thesis presents a theoretical cost model that describes when our prioritization and scheduling approach is worthwhile. The overall approach is validated on two industrial analytical applications in the area of energy management and predictive maintenance, showing that over 95% of the test failures are still reported back to the engineers while only 43% of the total available test cases are being executed

A Survey of Methods for Data Inclusion in System Dynamics Models

In 1980, Jay Forrester enumerated three types of data needed to develop the structure and decision rules in models: numerical, written and mental data, in increasing order of importance. While this prioritization is appropriate, it is numerical data that has experienced the most development in the 25 years since Forester made his enumeration. In this paper, we’ll focus on how numerical data can be incorporated into models when written and mental data are known, and survey the techniques for doing so

A survey on adaptive random testing

Random testing (RT) is a well-studied testing method that has been widely applied to the testing of many applications, including embedded software systems, SQL database systems, and Android applications. Adaptive random testing (ART) aims to enhance RT's failure-detection ability by more evenly spreading the test cases over the input domain. Since its introduction in 2001, there have been many contributions to the development of ART, including various approaches, implementations, assessment and evaluation methods, and applications. This paper provides a comprehensive survey on ART, classifying techniques, summarizing application areas, and analyzing experimental evaluations. This paper also addresses some misconceptions about ART, and identifies open research challenges to be further investigated in the future work

Condition Assessment Models for Sewer Pipelines

Underground pipeline system is a complex infrastructure system that has significant impact on social, environmental and economic aspects. Sewer pipeline networks are considered to be an extremely expensive asset. This study aims to develop condition assessment models for sewer pipeline networks. Seventeen factors affecting the condition of sewer network were considered for gravity pipelines in addition to the operating pressure for pressurized pipelines. Two different methodologies were adopted for models’ development. The first method by using an integrated Fuzzy Analytic Network Process (FANP) and Monte-Carlo simulation and the second method by using FANP, fuzzy set theory (FST) and Evidential Reasoning (ER). The models’ output is the assessed pipeline condition. In order to collect the necessary data for developing the models, questionnaires were distributed among experts in sewer pipelines in the state of Qatar. In addition, actual data for an existing sewage network in the state of Qatar was used to validate the models’ outputs. The “Ground Disturbance” factor was found to be the most influential factor followed by the “Location” factor with a weight of 10.6% and 9.3% for pipelines under gravity and 8.8% and 8.6% for pipelines under pressure, respectively. On the other hand, the least affecting factor was the “Length” followed by “Diameter” with weights of 2.2% and 2.5% for pipelines under gravity and 2.5% and 2.6% for pipelines under pressure. The developed models were able to satisfactorily assess the conditions of deteriorating sewer pipelines with an average validity of approximately 85% for the first approach and 86% for the second approach. The developed models are expected to be a useful tool for decision makers to properly plan for their inspections and provide effective rehabilitation of sewer networks.1)- NPRP grant # (NPRP6-357-2-150) from the QatarNational Research Fund (Member of Qatar Foundation) 2)-Tarek Zayed, Professor of Civil Engineeringat Concordia University for his support in the analysis part, the Public Works 3)-Authority of Qatar (ASHGAL) for their support in the data collection

R You Still Using Excel? The Advantages of Modern Software Tools for Health Technology Assessment

The Professional Society for Health Economics and Outcomes Research Economic models are used in health technology assessments (HTAs) to evaluate the cost-effectiveness of competing medical technologies and inform the efficient use of health care resources. Historically, these models have been developed with specialized commercial software (such as TreeAge) or more commonly with spreadsheet software (almost always Microsoft Excel). Although these tools may be sufficient for relatively simple analyses, they put unnecessary constraints on the analysis that may ultimately limit its credibility and relevance. In contrast, modern programming languages such as R, Python, Matlab, and Julia facilitate the development of models that are (i) clinically realistic, (ii) capable of quantifying decision uncertainty, (iii) transparent and reproducible, and (iv) reusable and adaptable. An HTA environment that encourages use of modern software can therefore help ensure that coverage and pricing decisions confer greatest possible benefit and capture all scientific uncertainty, thus enabling correct prioritization of future research

Seismic Reliability Assessment of Aging Highway Bridge Networks with Field Instrumentation Data and Correlated Failures. I: Methodology

The state-of-the-practice in seismic network reliability assessment of highway bridges often ignores bridge failure correlations imposed by factors such as the network topology, construction methods, and present-day condition of bridges, amongst others. Additionally, aging bridge seismic fragilities are typically determined using historical estimates of deterioration parameters. This research presents a methodology to estimate bridge fragilities using spatially interpolated and updated deterioration parameters from limited instrumented bridges in the network, while incorporating the impacts of overlooked correlation factors in bridge fragility estimates. Simulated samples of correlated bridge failures are used in an enhanced Monte Carlo method to assess bridge network reliability, and the impact of different correlation structures on the network reliability is discussed. The presented methodology aims to provide more realistic estimates of seismic reliability of aging transportation networks and potentially helps network stakeholders to more accurately identify critical bridges for maintenance and retrofit prioritization

RELIABILITY AND RISK ASSESSMENT OF NETWORKED URBAN INFRASTRUCTURE SYSTEMS UNDER NATURAL HAZARDS

Modern societies increasingly depend on the reliable functioning of urban infrastructure systems in the aftermath of natural disasters such as hurricane and earthquake events. Apart from a sizable capital for maintenance and expansion, the reliable performance of infrastructure systems under extreme hazards also requires strategic planning and effective resource assignment. Hence, efficient system reliability and risk assessment methods are needed to provide insights to system stakeholders to understand infrastructure performance under different hazard scenarios and accordingly make informed decisions in response to them. Moreover, efficient assignment of limited financial and human resources for maintenance and retrofit actions requires new methods to identify critical system components under extreme events. Infrastructure systems such as highway bridge networks are spatially distributed systems with many linked components. Therefore, network models describing them as mathematical graphs with nodes and links naturally apply to study their performance. Owing to their complex topology, general system reliability methods are ineffective to evaluate the reliability of large infrastructure systems. This research develops computationally efficient methods such as a modified Markov Chain Monte Carlo simulations algorithm for network reliability, and proposes a network reliability framework (BRAN: Bridge Reliability Assessment in Networks) that is applicable to large and complex highway bridge systems. Since the response of system components to hazard scenario events are often correlated, the BRAN framework enables accounting for correlated component failure probabilities stemming from different correlation sources. Failure correlations from non-hazard sources are particularly emphasized, as they potentially have a significant impact on network reliability estimates, and yet they have often been ignored or only partially considered in the literature of infrastructure system reliability. The developed network reliability framework is also used for probabilistic risk assessment, where network reliability is assigned as the network performance metric. Risk analysis studies may require prohibitively large number of simulations for large and complex infrastructure systems, as they involve evaluating the network reliability for multiple hazard scenarios. This thesis addresses this challenge by developing network surrogate models by statistical learning tools such as random forests. The surrogate models can replace network reliability simulations in a risk analysis framework, and significantly reduce computation times. Therefore, the proposed approach provides an alternative to the established methods to enhance the computational efficiency of risk assessments, by developing a surrogate model of the complex system at hand rather than reducing the number of analyzed hazard scenarios by either hazard consistent scenario generation or importance sampling. Nevertheless, the application of surrogate models can be combined with scenario reduction methods to improve even further the analysis efficiency. To address the problem of prioritizing system components for maintenance and retrofit actions, two advanced metrics are developed in this research to rank the criticality of system components. Both developed metrics combine system component fragilities with the topological characteristics of the network, and provide rankings which are either conditioned on specific hazard scenarios or probabilistic, based on the preference of infrastructure system stakeholders. Nevertheless, they both offer enhanced efficiency and practical applicability compared to the existing methods. The developed frameworks for network reliability evaluation, risk assessment, and component prioritization are intended to address important gaps in the state-of-the-art management and planning for infrastructure systems under natural hazards. Their application can enhance public safety by informing the decision making process for expansion, maintenance, and retrofit actions for infrastructure systems

Predicting deterioration rate of culvert structures utilizing a Markov model

A culvert is typically a hydraulic passage, normally placed perpendicular to the road alignment, which connects the upstream and downstream sections underneath an embankment, while also providing structural support for earth and traffic loads. The structural condition of culverts continues to deteriorate due to aging, limited maintenance budgets, and increased traffic loads. Maintaining the performance of culverts at acceptable levels is a priority for the U.S. Department of Transportation (DOT), and an effective maintenance of culvert structures can be greatly improved by introducing asset management practices. A priority list generated by traditional condition assessment might not provide optimum solutions, and benefits of culvert asset management practices can be maximized by incorporating prediction of deterioration trends. This dissertation includes the development of a decision making chart for culvert inspection, the development of a culvert rating methodology using the Analytic Hierarchy Process (AFIP) based on an expert opinion survey and the development of a Markovian model to predict the deterioration rate of culvert structures at the network level. The literature review is presented in three parts: culvert asset management systems in the U.S.; Non-destructive Technologies (NDT) for culvert inspection (concrete, metal, and thermoplastic culvert structures); and statistical approaches for estimating the deterioration rate for infrastructure. A review of available NDT methods was performed to identify methods applicable for culvert inspection. To identify practices currently used for culvert asset management, culvert inventory data requests were sent to 34 DOTs. The responses revealed that a relatively small number of DOTs manage their culvert assets using formal asset management systems and, while a number of DOTs have inventory databases, many do not have a methodology in place to convert them to priority lists. In addition, when making decisions, DOTs do not incorporate future deterioration rate information into the decision making process. The objective of this work was to narrow the gap between research and application. The culvert inventory database provides basic information support for culvert asset management. Preliminary data analysis of datasets provided by selected DOTs was performed to demonstrate the differences among them. An expert opinion survey using AHP was performed to confirm the weight of 23 factors, which was believed to contribute to the hydraulic & structural performance of culvert structures, so as to establish the culvert rating methodology. A homogenous Markov model, which was calibrated using the Metropolis-Hastings Algorithm, was utilized in the computation of the deterioration rate of culverts at the network level. A real world case study consisting of datasets of three highways inspected regularly by Oregon DOT is also presented. The performance of the model was validated using Pearson\u27s chi-square test

Stability and aggregation of ranked gene lists

Ranked gene lists are highly instable in the sense that similar measures of differential gene expression may yield very different rankings, and that a small change of the data set usually affects the obtained gene list considerably. Stability issues have long been under-considered in the literature, but they have grown to a hot topic in the last few years, perhaps as a consequence of the increasing skepticism on the reproducibility and clinical applicability of molecular research findings. In this article, we review existing approaches for the assessment of stability of ranked gene lists and the related problem of aggregation, give some practical recommendations, and warn against potential misuse of these methods. This overview is illustrated through an application to a recent leukemia data set using the freely available Bioconductor package GeneSelector