Flaky tests cause significant problems as they can interrupt automated build
processes that rely on all tests succeeding and undermine the trustworthiness
of tests. Numerous causes of test flakiness have been identified, and program
analyses exist to detect such tests. Typically, these methods produce advice to
developers on how to refactor tests in order to make test outcomes
deterministic. We argue that one source of flakiness is the lack of assumptions
that precisely describe under which circumstances a test is meaningful. We
devise a sanitisation technique that can isolate f laky tests quickly by
inferring such assumptions on-the-fly, allowing automated builds to proceed as
flaky tests are ignored. We demonstrate this approach for Java and Groovy
programs by implementing it as extensions for three popular testing frameworks
(JUnit4, JUnit5 and Spock) that can transparently inject the inferred
assumptions. If JUnit5 is used, those extensions can be deployed without
refactoring project source code. We demonstrate and evaluate the utility of our
approach using a set of six popular real-world programs, addressing known test
flakiness issues in these programs caused by dependencies of tests on network
availability. We find that our method effectively sanitises failures induced by
network connectivity problems with high precision and recall.Comment: to appear at IEEE International Working Conference on Source Code
Analysis and Manipulation (SCAM