Testing web enabled simulation at scale using metamorphic testing

We report on Facebook's deployment of MIA (Metamorphic Interaction Automaton). MIA is used to test Facebook's Web Enabled Simulation, built on a web infrastructure of hundreds of millions of lines of code. MIA tackles the twin problems of test flakiness and the unknowable oracle problem. It uses metamorphic testing to automate continuous integration and regression test execution. MIA also plays the role of a test bot, automatically commenting on all relevant changes submitted for code review. It currently uses a suite of over 40 metamorphic test cases. Even at this extreme scale, a non-trivial metamorphic test suite subset yields outcomes within 20 minutes (sufficient for continuous integration and review processes). Furthermore, our offline mode simulation reduces test flakiness from approximately 50% (of all online tests) to 0% (offline). Metamorphic testing has been widely-studied for 22 years. This paper is the first reported deployment into an industrial continuous integration system

Performance-Driven Metamorphic Testing of Cyber-Physical Systems

Cyber-physical systems (CPSs) are a new generation of systems, which integrate software with physical processes. The increasing complexity of these systems, combined with the un certainty in their interactions with the physical world, makes the definition of effective test oracles especially challenging, facing the well-known test oracle problem. Metamorphic testing has shown great potential to alleviate the test oracle problem by exploiting the relations among the inputs and outputs of different executions of the system, so-called metamorphic relations (MRs). In this article, we propose an MR pattern called PV for the identification of performance-driven MRs, and we show its applicability in two CPSs from different domains, which are automated navigation systems and elevator control systems. For the evaluation, we as sessed the effectiveness of this approach for detecting failures in an open-source simulation-based autonomous navigation system, as well as in an industrial case study from the elevation domain. We derive concrete MRs based on the PV pattern for both case studies, and we evaluate their effectiveness with seeded faults. Results show that the approach is effective at detecting over 88% of the seeded faults, while keeping the ratio of FPs at 4% or lower.European Union's Horizon 2020 Research and Innovation Programme (Grant Number: 871319)Junta de Andalucía US-1264651 (APOLO)Junta de Andalucía P18-FR-2895 (EKIPMENT-PLUS)Ministerio de Ciencia e Innovación RTI2018-101204-B-C21 (HORATIO)Mondragon Unibertsitatea IT1519-2