1 research outputs found
JEST: N+1-version Differential Testing of Both JavaScript Engines and Specification
Modern programming follows the continuous integration (CI) and continuous
deployment (CD) approach rather than the traditional waterfall model. Even the
development of modern programming languages uses the CI/CD approach to swiftly
provide new language features and to adapt to new development environments.
Unlike in the conventional approach, in the modern CI/CD approach, a language
specification is no more the oracle of the language semantics because both the
specification and its implementations can co-evolve. In this setting, both the
specification and implementations may have bugs, and guaranteeing their
correctness is non-trivial.
In this paper, we propose a novel N+1-version differential testing to resolve
the problem. Unlike the traditional differential testing, our approach consists
of three steps: 1) to automatically synthesize programs guided by the syntax
and semantics from a given language specification, 2) to generate conformance
tests by injecting assertions to the synthesized programs to check their final
program states, 3) to detect bugs in the specification and implementations via
executing the conformance tests on multiple implementations, and 4) to localize
bugs on the specification using statistical information. We actualize our
approach for the JavaScript programming language via JEST, which performs
N+1-version differential testing for modern JavaScript engines and ECMAScript,
the language specification describing the syntax and semantics of JavaScript in
a natural language. We evaluated JEST with four JavaScript engines that support
all modern JavaScript language features and the latest version of ECMAScript
(ES11, 2020). JEST automatically synthesized 1,700 programs that covered 97.78%
of syntax and 87.70% of semantics from ES11. Using the assertion-injection, it
detected 44 engine bugs in four engines and 27 specification bugs in ES11.Comment: 12 pages, 5 figures, 3 tables, In Proceedings of the ACM/IEEE 43rd
International Conference on Software Engineering (ICSE 2021