4 research outputs found
Using Machine Learning to Generate Test Oracles: A Systematic Literature Review
Machine learning may enable the automated generation of test oracles. We have characterized emerging research in this area through a systematic literature review examining oracle types, researcher goals, the ML techniques applied, how the generation process was assessed, and the open research challenges in this emerging field.Based on a sample of 22 relevant studies, we observed that ML algorithms generated test verdict, metamorphic relation, and - most commonly - expected output oracles. Almost all studies employ a supervised or semi-supervised approach, trained on labeled system executions or code metadata - including neural networks, support vector machines, adaptive boosting, and decision trees. Oracles are evaluated using the mutation score, correct classifications, accuracy, and ROC. Work-to-date show great promise, but there are significant open challenges regarding the requirements imposed on training data, the complexity of modeled functions, the ML algorithms employed - and how they are applied - the benchmarks used by researchers, and replicability of the studies. We hope that our findings will serve as a roadmap and inspiration for researchers in this field
Using Machine Learning to Generate Test Oracles: A Systematic Literature Review
Machine learning may enable the automated generation of test oracles. We have
characterized emerging research in this area through a systematic literature
review examining oracle types, researcher goals, the ML techniques applied, how
the generation process was assessed, and the open research challenges in this
emerging field.
Based on a sample of 22 relevant studies, we observed that ML algorithms
generated test verdict, metamorphic relation, and - most commonly - expected
output oracles. Almost all studies employ a supervised or semi-supervised
approach, trained on labeled system executions or code metadata - including
neural networks, support vector machines, adaptive boosting, and decision
trees. Oracles are evaluated using the mutation score, correct classifications,
accuracy, and ROC. Work-to-date show great promise, but there are significant
open challenges regarding the requirements imposed on training data, the
complexity of modeled functions, the ML algorithms employed - and how they are
applied - the benchmarks used by researchers, and replicability of the studies.
We hope that our findings will serve as a roadmap and inspiration for
researchers in this field.Comment: Pre-print. Article accepted to 1st International Workshop on Test
Oracles at ESEC/FSE 202
Learning to Encode and Classify Test Executions
The challenge of automatically determining the correctness of test executions
is referred to as the test oracle problem and is one of the key remaining
issues for automated testing. The goal in this paper is to solve the test
oracle problem in a way that is general, scalable and accurate.
To achieve this, we use supervised learning over test execution traces. We
label a small fraction of the execution traces with their verdict of pass or
fail. We use the labelled traces to train a neural network (NN) model to learn
to distinguish runtime patterns for passing versus failing executions for a
given program. Our approach for building this NN model involves the following
steps, 1. Instrument the program to record execution traces as sequences of
method invocations and global state, 2. Label a small fraction of the execution
traces with their verdicts, 3. Designing a NN component that embeds information
in execution traces to fixed length vectors, 4. Design a NN model that uses the
trace information for classification, 5. Evaluate the inferred classification
model on unseen execution traces from the program.
We evaluate our approach using case studies from different application
domains: 1. Module from Ethereum Blockchain, 2. Module from PyTorch deep
learning framework, 3. Microsoft SEAL encryption library components, 4. Sed
stream editor, 5. Value pointer library and 6. Nine network protocols from
Linux packet identifier, L7-Filter. We found the classification models for all
subject programs resulted in high precision, recall and specificity, over 95%,
while only training with an average 9% of the total traces. Our experiments
show that the proposed neural network model is highly effective as a test
oracle and is able to learn runtime patterns to distinguish passing and failing
test executions for systems and tests from different application domains