Towards incremental component compatibility testing

Software components are increasingly assembled from other components. Each component may further depend on oth-ers, and each may have multiple active versions. The to-tal number of configurations—combinations of components and their versions—deployed by end users can be very large. Component developers, therefore, spend considerable time and effort doing compatibility testing – determining whether their components can be built correctly for all deployed con-figurations. In previous work we developed Rachet to sup-port large-scale compatibility testing of components. In this paper, we describe and evaluate methods to enable Rachet to perform incremental compatibility testing. We de-scribe algorithms to compute differences in component com-patibilities between current and previous component builds, a formal test adequacy criterion based on covering the differ-ences, and cache-aware configuration sampling and testing methods that attempt to reuse effort from previous testing sessions. We evaluate our approach using the 5-year evo-lution history of a scientific middleware component. Our results show significant performance improvements over Ra-chet’s previous retest-all approach, making the process of compatibility testing practical for evolving components