2,811 research outputs found

    Fault Detection Effectiveness of Metamorphic Relations Developed for Testing Supervised Classifiers

    Full text link
    In machine learning, supervised classifiers are used to obtain predictions for unlabeled data by inferring prediction functions using labeled data. Supervised classifiers are widely applied in domains such as computational biology, computational physics and healthcare to make critical decisions. However, it is often hard to test supervised classifiers since the expected answers are unknown. This is commonly known as the \emph{oracle problem} and metamorphic testing (MT) has been used to test such programs. In MT, metamorphic relations (MRs) are developed from intrinsic characteristics of the software under test (SUT). These MRs are used to generate test data and to verify the correctness of the test results without the presence of a test oracle. Effectiveness of MT heavily depends on the MRs used for testing. In this paper we have conducted an extensive empirical study to evaluate the fault detection effectiveness of MRs that have been used in multiple previous studies to test supervised classifiers. Our study uses a total of 709 reachable mutants generated by multiple mutation engines and uses data sets with varying characteristics to test the SUT. Our results reveal that only 14.8\% of these mutants are detected using the MRs and that the fault detection effectiveness of these MRs do not scale with the increased number of mutants when compared to what was reported in previous studies.Comment: 8 pages, AITesting 201

    Using Machine Learning to Generate Test Oracles: A Systematic Literature Review

    Get PDF
    Machine learning may enable the automated generation of test oracles. We have characterized emerging research in this area through a systematic literature review examining oracle types, researcher goals, the ML techniques applied, how the generation process was assessed, and the open research challenges in this emerging field.Based on a sample of 22 relevant studies, we observed that ML algorithms generated test verdict, metamorphic relation, and - most commonly - expected output oracles. Almost all studies employ a supervised or semi-supervised approach, trained on labeled system executions or code metadata - including neural networks, support vector machines, adaptive boosting, and decision trees. Oracles are evaluated using the mutation score, correct classifications, accuracy, and ROC. Work-to-date show great promise, but there are significant open challenges regarding the requirements imposed on training data, the complexity of modeled functions, the ML algorithms employed - and how they are applied - the benchmarks used by researchers, and replicability of the studies. We hope that our findings will serve as a roadmap and inspiration for researchers in this field

    Using Machine Learning to Generate Test Oracles: A Systematic Literature Review

    Get PDF
    Machine learning may enable the automated generation of test oracles. We have characterized emerging research in this area through a systematic literature review examining oracle types, researcher goals, the ML techniques applied, how the generation process was assessed, and the open research challenges in this emerging field. Based on a sample of 22 relevant studies, we observed that ML algorithms generated test verdict, metamorphic relation, and - most commonly - expected output oracles. Almost all studies employ a supervised or semi-supervised approach, trained on labeled system executions or code metadata - including neural networks, support vector machines, adaptive boosting, and decision trees. Oracles are evaluated using the mutation score, correct classifications, accuracy, and ROC. Work-to-date show great promise, but there are significant open challenges regarding the requirements imposed on training data, the complexity of modeled functions, the ML algorithms employed - and how they are applied - the benchmarks used by researchers, and replicability of the studies. We hope that our findings will serve as a roadmap and inspiration for researchers in this field.Comment: Pre-print. Article accepted to 1st International Workshop on Test Oracles at ESEC/FSE 202

    Mapping soil water holding capacity over large areas to predict the potential production of forest stands

    Get PDF
    International audienceEcological studies need environmental descriptors to establish the response of species or communities to ecological conditions. Soil water resource is an important factor but is poorly used by plant ecologists because of the lack of accessible data. We explore whether a large number of plots with basic soil information collected within the framework of forest inventories allows the soil water holding capacity (SWHC) to be mapped with enough accuracy to predict tree species growth over large areas. We first compared the performance of available pedotransfer functions (PTFs) and showed significant differences in the prediction quality of SWHC between the PTFs selected. We also showed that the most efficient class PTFs and continuous PTFs compared had similar performance, but there was a significant reduction in efficiency when they were applied to soils different from those used to calibrate them. With a root mean squared error (RMSE) of 0.046 cm3 cm-3 (n = 227 horizons), we selected the Al Majou class PTFs to predict the SWHC in the soil horizons described in every plot, thus allowing 84% of SWHC variance to be explained in soils free of stone (n = 63 plots). Then, we estimated the soil water holding capacity by integrating the stone content collected at the soil pit scale (SWHC') and both the stone content at the soil pit scale and rock outcrop at the plot scale (SWHC") for the 100.307 forest plots recorded in France within the framework of forest inventories. The SWHC" values were interpolated by kriging to produce a map with 1 km² cell size, with a wider resolution leading to a decrease in map accuracy. The SWHC" given by the map ranged from 0 to 148 mm for a soil down to 1 m depth. The RMSE between map values and plot estimates was 33.9 mm, the best predictions being recorded for soils developed on marl, clay, and hollow silicate rocks, and in flat areas. Finally, the ability of SWHC' and SWHC" to predict Ecological studies need environmental descriptors to establish the response of species or communities to ecological conditions. Soil water resource is an important factor but is poorly used by plant ecologists because of the lack of accessible data. We explore whether a large number of plots with basic soil information collected within the framework of forest inventories allows the soil water holding capacity (SWHC) to be mapped with enough accuracy to predict tree species growth over large areas. We first compared the performance of available pedotransfer functions (PTFs) and showed significant differences in the prediction quality of SWHC between the PTFs selected. We also showed that the most efficient class PTFs and continuous PTFs compared had similar performance, but there was a significant reduction in efficiency when they were applied to soils different from those used to calibrate them. With a root mean squared error (RMSE) of 0.046 cm3 cm-3 (n = 227 horizons), we selected the Al Majou class PTFs to predict the SWHC in the soil horizons described in every plot, thus allowing 84% of SWHC variance to be explained in soils free of stone (n = 63 plots). Then, we estimated the soil water holding capacity by integrating the stone content collected at the soil pit scale (SWHC') and both the stone content at the soil pit scale and rock outcrop at the plot scale (SWHC") for the 100.307 forest plots recorded in France within the framework of forest inventories. The SWHC" values were interpolated by kriging to produce a map with 1 km² cell size, with a wider resolution leading to a decrease in map accuracy. The SWHC" given by the map ranged from 0 to 148 mm for a soil down to 1 m depth. The RMSE between map values and plot estimates was 33.9 mm, the best predictions being recorded for soils developed on marl, clay, and hollow silicate rocks, and in flat areas. Finally, the ability of SWHC' and SWHC" to predict height growth for Fagus sylvatica, Picea abies and Quercus petraea is discussed. We show a much better predictive ability for SWHC" compared to SWHC'. The values of SWHC" extracted from the map were significantly related to tree height growth. They explained 10.7% of the height growth index variance for Fagus sylvatica (n = 866), 14.1% for Quercus petraea (n = 877) and 10.3% for Picea abies (n = 2067). The proportions of variance accounted by SWHC" were close to those recorded with SWHC" values estimated from the plots (11.5, 11.7, and 18.6% for Fagus sylvatica, Quercus petraea and Picea abies, respectively). We conclude that SWHC" can be mapped using basic soil parameters collected from plots, the predictive ability of the map and of data derived from the plot being close. Thus, the map could be used just as well for small areas as for large areas, directly or indirectly through water balance indices, to predict forest growth and thus production, today or in the future, in the context of an increasing drought period linked to a global change of climatic conditions

    A Survey on Metamorphic Testing

    Full text link

    A Survey on Metamorphic Testing

    Get PDF
    A test oracle determines whether a test execution reveals a fault, often by comparing the observed program output to the expected output. This is not always practical, for example when a program's input-output relation is complex and difficult to capture formally. Metamorphic testing provides an alternative, where correctness is not determined by checking an individual concrete output, but by applying a transformation to a test input and observing how the program output “morphs” into a different one as a result. Since the introduction of such metamorphic relations in 1998, many contributions on metamorphic testing have been made, and the technique has seen successful applications in a variety of domains, ranging from web services to computer graphics. This article provides a comprehensive survey on metamorphic testing: It summarises the research results and application areas, and analyses common practice in empirical studies of metamorphic testing as well as the main open challenges

    A Survey on Metamorphic Testing

    Get PDF
    A test oracle determines whether a test execution reveals a fault, often by comparing the observed program output to the expected output. This is not always practical, for example when a program’s input-output relation is complex and difficult to capture formally. Metamorphic testing provides an alternative, where correctness is not determined by checking an individual concrete output, but by applying a transformation to a test input and observing how the program output “morphs” into a different one as a result. Since the introduction of such metamorphic relations in 1998, many contributions on metamorphic testing have been made, and the technique has seen successful applications in a variety of domains, ranging from web services to computer graphics. This article provides a comprehensive survey on metamorphic testing: It summarises the research results and application areas, and analyses common practice in empirical studies of metamorphic testing as well as the main open challenges.European Commission (FEDER)Spanish Govermen

    Automatic System Testing of Programs without Test Oracles

    Get PDF
    Metamorphic testing has been shown to be a simple yet effective technique in addressing the quality assurance of applications that do not have test oracles, i.e., for which it is difficult or impossible to know what the correct output should be for arbitrary input. In metamorphic testing, existing test case input is modified to produce new test cases in such a manner that, when given the new input, the application should produce an output that can be easily be computed based on the original output. That is, if input x produces output f (x), then we create input x' such that we can predict f (x') based on f(x); if the application does not produce the expected output, then a defect must exist, and either f (x) or f (x') (or both) is wrong. In practice, however, metamorphic testing can be a manually intensive technique for all but the simplest cases. The transformation of input data can be laborious for large data sets, or practically impossible for input that is not in human-readable format. Similarly, comparing the outputs can be error-prone for large result sets, especially when slight variations in the results are not actually indicative of errors (i.e., are false positives), for instance when there is non-determinism in the application and multiple outputs can be considered correct. In this paper, we present an approach called Automated Metamorphic System Testing. This involves the automation of metamorphic testing at the system level by checking that the metamorphic properties of the entire application hold after its execution. The tester is able to easily set up and conduct metamorphic tests with little manual intervention, and testing can continue in the field with minimal impact on the user. Additionally, we present an approach called Heuristic Metamorphic Testing which seeks to reduce false positives and address some cases of non-determinism. We also describe an implementation framework called Amsterdam, and present the results of empirical studies in which we demonstrate the effectiveness of the technique on real-world programs without test oracles
    • …
    corecore