Effective and scalable software compatibility testing

Abstract

Today’s software systems are typically composed of multiple components, each with different versions. Software compat-ibility testing is a quality assurance task aimed at ensuring that multi-component based systems build and/or execute correctly across all their versions ’ combinations, or config-urations. Because there are complex and changing interde-pendencies between components and their versions, and be-cause there are such a large number of configurations, it is generally infeasible to test all potential configurations. Con-sequently, in practice, compatibility testing examines only a handful of default or popular configurations to detect prob-lems; as a result costly errors can and do escape to the field. This paper presents a new approach to compatibility test-ing, called Rachet. We formally model the entire config-uration space for software systems and use the model to generate test plans to sample a portion of the space. In this paper, we test all direct dependencies between components and execute the test plan efficiently in parallel. We present empirical results obtained by applying our approach to two large-scale scientific middleware systems. The results show that for these systems Rachet scaled well and discovered in-compatibilities between components, and that testing only direct dependences did not compromise test quality