4 research outputs found
Using Machine Learning to Generate Test Oracles: A Systematic Literature Review
Machine learning may enable the automated generation of test oracles. We have characterized emerging research in this area through a systematic literature review examining oracle types, researcher goals, the ML techniques applied, how the generation process was assessed, and the open research challenges in this emerging field.Based on a sample of 22 relevant studies, we observed that ML algorithms generated test verdict, metamorphic relation, and - most commonly - expected output oracles. Almost all studies employ a supervised or semi-supervised approach, trained on labeled system executions or code metadata - including neural networks, support vector machines, adaptive boosting, and decision trees. Oracles are evaluated using the mutation score, correct classifications, accuracy, and ROC. Work-to-date show great promise, but there are significant open challenges regarding the requirements imposed on training data, the complexity of modeled functions, the ML algorithms employed - and how they are applied - the benchmarks used by researchers, and replicability of the studies. We hope that our findings will serve as a roadmap and inspiration for researchers in this field
Application of Quantum Extreme Learning Machines for QoS Prediction of Elevators' Software in an Industrial Context
Quantum Extreme Learning Machine (QELM) is an emerging technique that
utilizes quantum dynamics and an easy-training strategy to solve problems such
as classification and regression efficiently. Although QELM has many potential
benefits, its real-world applications remain limited. To this end, we present
QELM's industrial application in the context of elevators, by proposing an
approach called QUELL. In QUELL, we use QELM for the waiting time prediction
related to the scheduling software of elevators, with applications for software
regression testing, elevator digital twins, and real-time performance
prediction. The scheduling software has been implemented by our industrial
partner Orona, a globally recognized leader in elevator technology. We
demonstrate that QUELL can efficiently predict waiting times, with prediction
quality significantly better than that of classical ML models employed in a
state-of-the-practice approach. Moreover, we show that the prediction quality
of QUELL does not degrade when using fewer features. Based on our industrial
application, we further provide insights into using QELM in other applications
in Orona, and discuss how QELM could be applied to other industrial
applications
Using Machine Learning to Generate Test Oracles: A Systematic Literature Review
Machine learning may enable the automated generation of test oracles. We have
characterized emerging research in this area through a systematic literature
review examining oracle types, researcher goals, the ML techniques applied, how
the generation process was assessed, and the open research challenges in this
emerging field.
Based on a sample of 22 relevant studies, we observed that ML algorithms
generated test verdict, metamorphic relation, and - most commonly - expected
output oracles. Almost all studies employ a supervised or semi-supervised
approach, trained on labeled system executions or code metadata - including
neural networks, support vector machines, adaptive boosting, and decision
trees. Oracles are evaluated using the mutation score, correct classifications,
accuracy, and ROC. Work-to-date show great promise, but there are significant
open challenges regarding the requirements imposed on training data, the
complexity of modeled functions, the ML algorithms employed - and how they are
applied - the benchmarks used by researchers, and replicability of the studies.
We hope that our findings will serve as a roadmap and inspiration for
researchers in this field.Comment: Pre-print. Article accepted to 1st International Workshop on Test
Oracles at ESEC/FSE 202
The Integration of Machine Learning into Automated Test Generation: A Systematic Mapping Study
Context: Machine learning (ML) may enable effective automated test
generation.
Objective: We characterize emerging research, examining testing practices,
researcher goals, ML techniques applied, evaluation, and challenges.
Methods: We perform a systematic mapping on a sample of 102 publications.
Results: ML generates input for system, GUI, unit, performance, and
combinatorial testing or improves the performance of existing generation
methods. ML is also used to generate test verdicts, property-based, and
expected output oracles. Supervised learning - often based on neural networks -
and reinforcement learning - often based on Q-learning - are common, and some
publications also employ unsupervised or semi-supervised learning.
(Semi-/Un-)Supervised approaches are evaluated using both traditional testing
metrics and ML-related metrics (e.g., accuracy), while reinforcement learning
is often evaluated using testing metrics tied to the reward function.
Conclusion: Work-to-date shows great promise, but there are open challenges
regarding training data, retraining, scalability, evaluation complexity, ML
algorithms employed - and how they are applied - benchmarks, and replicability.
Our findings can serve as a roadmap and inspiration for researchers in this
field.Comment: Under submission to Software Testing, Verification, and Reliability
journal. (arXiv admin note: text overlap with arXiv:2107.00906 - This is an
earlier study that this study extends