Simulation platforms facilitate the development of emerging Cyber-Physical
Systems (CPS) like self-driving cars (SDC) because they are more efficient and
less dangerous than field operational test cases. Despite this, thoroughly
testing SDCs in simulated environments remains challenging because SDCs must be
tested in a sheer amount of long-running test cases. Past results on software
testing optimization have shown that not all the test cases contribute equally
to establishing confidence in test subjects' quality and reliability, and the
execution of "safe and uninformative" test cases can be skipped to reduce
testing effort. However, this problem is only partially addressed in the
context of SDC simulation platforms. In this paper, we investigate test
selection strategies to increase the cost-effectiveness of simulation-based
testing in the context of SDCs. We propose an approach called SDC-Scissor (SDC
coSt-effeCtIve teSt SelectOR) that leverages Machine Learning (ML) strategies
to identify and skip test cases that are unlikely to detect faults in SDCs
before executing them.
Our evaluation shows that SDC-Scissor outperforms the baselines. With the
Logistic model, we achieve an accuracy of 70%, a precision of 65%, and a recall
of 80% in selecting tests leading to a fault and improved testing
cost-effectiveness. Specifically, SDC-Scissor avoided the execution of 50% of
unnecessary tests as well as outperformed two baseline strategies.
Complementary to existing work, we also integrated SDC-Scissor into the context
of an industrial organization in the automotive domain to demonstrate how it
can be used in industrial settings.Comment: arXiv admin note: substantial text overlap with arXiv:2111.0466