Comparing the effectiveness of testing methods in improving programs: the effect of variations in program quality

We compare the efficacy of different testing methods for improving the reliability of software. Specifically, we use modelling to compare “operational” testing, in which test cases are chosen according to their probability of occurring in actual use of the software, against “debug” testing methods, in which the testers look for test cases which they consider likely to cause failure, or that satisfy some coverage criterion. We base our comparisons on the reliability reached by the program at the end of testing. Differently from previous studies, we consider the probability distribution of the achieved reliability, and thus the probability of satisfying specific requirements, rather than just the average reliability achieved. We take account of two sources of variation. The variation between the actual test histories that are possible for a given program and a given test method: and the fact that different programs start testing with different faults and initial reliability levels. By necessity, we use very simplified models of reality. Yet, we can show some interesting conclusions with important practical consequences. In general, there are stronger arguments in favor of operational testing than previous studies have show

FAUSTA: Scaling Dynamic Analysis with Traffic Generation at WhatsApp

We introduce Fausta, an algorithmic traffic gener-ation platform that enables analysis and testing at scale. Fausta has been deployed at Meta to analyze and test the WhatsApp plat-form infrastructure since September 2020, enabling WhatsApp developers to deploy reliable code changes to a code base of millions of lines of code, supporting over 2 billion users who rely on WhatsApp for their daily communications. Fausta covers expected and unexpected program behaviors in a privacy-safe controlled environment to support multiple use cases such as reliability testing, privacy analysis and performance regression detection. It currently supports three different algorithmic input generation strategies, each of which construct realistic backend server traffic that closely simulates production data, without replaying any real user data. Fausta has been deployed and closely integrated into the WhatsApp continuous integration process, catching bugs in development before they hit production. We report on the development and deployment of Fausta's reliability use case between September 2020 and August 2021. During this period it has found 1,876 unique reliability issues, with a fix rate of 74%, indicating a high degree of true positive fault revelation. We also report on the distribution of fault types revealed by Fausta, and the correlation between coverage and faults found. Overall, we do find evidence that higher coverage is correlated with fault revelation

Proportional sampling strategy: A compendium and some insights

There have been numerous studies on the effectiveness of partition and random testing. In particular, the proportional sampling (PS) strategy has been proved, under certain conditions, to be the only form of partition testing that outperforms random testing regardless of where the failure-causing inputs are. This paper provides an integrated synthesis and overview of our recent studies on the PS strategy and its related work. Through this synthesis, we offer a perspective that properly interprets the results obtained so far, and present some of the interesting issues involved and new insights obtained during the course of this research. © 2001 Elsevier Science Inc. All rights reserved.postprin

On Input Profile Selection For Software Testing

This paper analyzes the effect of input pro le selection on software testing using the concept of fault detectability profile. It shows that optimality of the input profile during testing depends on factors such as the planned testing effort and the error detectability pro le. To achieve ultra-reliable software, selecting test input uniformly among different input domains is preferred. On the other hand, if testing effort is limited due to cost or schedule constraints, one should test only the highly used in-put domains. Use of operational pro le is also needed for accurate determination of operational reliability

On Input Profile Selection For Software Testing *

This paper analyzes the effect of input profile se-lection on software testing using the concept of fault detectability profile. It shows that optimality of the input profile during testing depends on factors such as the planned testing effort and the fault detectabil-ity profile. To achieve ultra-reliable software, selecting test input uniformly among different input domains is preferred. On the other hand, if testing effort is lim-ited due to cost or schedule constraints, one should test only the highly used input domains. Use of opera-tional profile is also needed for accurate determination of operational reliability.