This document is an extended abstract of an IEEE Software paper, "Metamorphic Testing: Testing the Untestable," presented as a J1C2 (Journal publication first, Conference presentation following) at the IEEE Computer Society signature conference on Computers, Software and Applications (COMPSAC 2019), hosted by Marquette University, Milwaukee, Wisconsin, USA. © 2019 IEEE

Chen, T.Y.

Segura, Sergio

Towey, Dave

Zhou, Zhi Quan

English

Crossref

An Extended Abstract of "Metamorphic Testing: Testing the Untestable"

Nottingham ePrints

                             An Extended Abstract of “Metamorphic Testing: Testing the Untestable”  Segura, S., Towey, D., Zhou, Z.Q., Chen, T.Y.   University of Nottingham Ningbo China, 199 Taikang East Road, Ningbo, 315100, China  First published 2019  This work is made available under the terms of the Creative Commons Attribution 4.0 International License: http://creativecommons.org/licenses/by/4.0    The work is licenced to the University of Nottingham Ningbo China under the Global University Publication Licence: https://www.nottingham.edu.cn/en/library/documents/research-support/global-university-publications-licence.pdf  XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE *Dave Towey is the corresponding author. An Extended Abstract of “Metamorphic Testing: Testing the Untestable”  Sergio Segura  Department of Computer Languages and Systems, University of Seville, Spain sergiosegura@us.es  Dave Towey* School of Computer Science, University of Nottingham Ningbo China, China dave.towey@nottingham.edu.cn  Zhi Quan Zhou School of Computing and Information Technology, University of Wollongong, Australia zhiquan@uow.edu.au  T.Y. Chen Department of Computer Science and Software Engineering, Swinburne University of Technology, Australia tychen@swin.edu.au     Abstract—This document is an extended abstract of an IEEE Software paper, “Metamorphic Testing: Testing the Untestable,” presented as a J1C2 (Journal publication first, Conference presentation following) at the IEEE Computer Society signature conference on Computers, Software and Applications (COMPSAC 2019), hosted by Marquette University, Milwaukee, Wisconsin, USA. Keywords—software testing, metamorphic testing, oracle problem, test case generation, untestable systems I. INTRODUCTION This is an extended abstract of the paper “Metamorphic Testing: Testing the Untestable” [1]. II. SUMMARY OF THE KEY RESULTS AND CONTRIBUTIONS  Metamorphic testing (MT) has, over the course of its twenty-year history, become a well-recognised approach to software testing. MT's popularity has been growing very rapidly in recent years, prompted, partly, by the recognition of its potential application to test “untestable” systems — systems where determining the correctness of output or behaviour for a given input is not possible or is impractical. Some examples of such systems include compilers, search engines, machine learning systems, simulators, and autonomous vehicles. The developers of these systems are increasingly turning to MT to address their testing challenges. Our paper serves as a hands-on entry point for newcomers to metamorphic testing, describing examples, possible applications, and current limitations, providing readers with the basics for the application of the technique in their own projects. III. RECENT MT DEVELOPMENTS  It is a very exciting time for the metamorphic testing community. Over the past few months we have seen an incredible uptake in researchers and industry applying the MT techniques to the testing of software and systems previously deemed “untestable.” The growing ubiquity of Artificial Intelligence (AI) and especially Machine Learning (ML) systems, has created an incredible opportunity for innovation and evolution of our current approaches to software quality assurance and testing. MT, now twenty years old, is fast-becoming the first choice for testers of these systems. Since the acceptance for publication of our paper (August 2018) [1], a number of new and exciting directions for MT have come to light, including (but not limited to): metamorphic exploration (an MT-inspired approach that helps users, including testers, better understand and use systems) [2]; MT’s application to post-quantum cryptography, as explored at the Loyola University and the American National Institute of Standards and Technology (NIST) [3]; simulation validation using MT [4]; metamorphic robustness testing (revealing hidden defects in the citation database systems Scopus and Web of Science, hence challenging the validity of citation statistics and journal impact factors) [5]; and a reported technique for detecting fatal software faults in the LiDAR obstacle-perception module of self-driving cars, with alarming results reported eight days before Uber’s deadly crash in Tempe, Arizona, USA, in March 2018 [6].  Our J1C2 presentation at COMPSAC introduces participants to MT, illustrating how it can be adapted and applied to many different situations (including both “untestable” and testable systems), and outlining some of the most recent and future advances in MT.  ACKNOWLEDGMENT Dave Towey acknowledges the support from the Natural Science Foundation of China (project code 61872167).  REFERENCES [1] S. Segura, D. Towey, Z.Q. Zhou and T.Y. Chen, “Metamorphic Testing: Testing the Untestable,” IEEE Software. [Online] Available: https://doi.org/10.1109/MS.2018.2875968 [2] Z.Q. Zhou, L. Sun, T.Y. Chen, and D. Towey, “Metamorphic Relations for Enhancing System Understanding and Use,” IEEE Transactions on Software Engineering. [Online] Available:  https://doi.org/10.1109/TSE.2018.2876433 [3] S. Pugh, M.S. Raunak, R. Kuhn and R. Kacker, “Systematic Testing of Post-Quantum Cryptographic Implementations Using MT,” in Proceedings of the IEEE/ACM 4th International Workshop on Metamorphic Testing (ICSE MET 2019), in conjunction with the 41st International Conference on Software Engineering (ICSE), 2019.  [4] M. Olsen and M. Raunak, “Increasing Validity of Simulation Models Through Metamorphic Testing,” IEEE Transactions on Reliability, vol. 68, no. 1, pp. 91-108, March 2019. [Online] Available:  https://doi.org/10.1109/TR.2018.2850315   [5] Z.Q. Zhou, T.H. Tse, and M. Witheridge “Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors,” IEEE Transactions on Software Engineering. [Online] Available:  https://doi.org/10.1109/TSE.2019.2915065 [6] Z.Q. Zhou and L. Sun, “Metamorphic testing of driverless cars,” Communications of the ACM, vol. 62, no. 3, pp. 61–67, March 2019. [Online] Available: https://doi.org/10.1145/3241979  

An extended abstract of "Metamorphic testing: testing the untestable"

This document is an extended abstract of an IEEE
Software paper, “Metamorphic Testing: Testing the Untestable,”
presented as a J1C2 (Journal publication first, Conference
presentation following) at the IEEE Computer Society signature
conference on Computers, Software and Applications
(COMPSAC 2019), hosted by Marquette University, Milwaukee,
Wisconsin, USA

An extended abstract of "Metamorphic testing: testing the untestable"

Abstract

Similar works

Full text

Available Versions

Crossref

Nottingham ePrints

idUS. Depósito de Investigación Universidad de Sevilla

Nottingham eTheses