GUI is a bridge connecting user and application. Existing GUI testing tasks
can be categorized into two groups: functionality testing and compatibility
testing. While the functionality testing focuses on detecting application
runtime bugs, the compatibility testing aims at detecting bugs resulting from
device or platform difference. To automate testing procedures and improve
testing efficiency, previous works have proposed dozens of tools. To evaluate
these tools, in functionality testing, researchers have published testing
benchmarks. Comparatively, in compatibility testing, the question of ``Do
existing methods indeed effectively assist test cases replay?'' is not well
answered. To answer this question and advance the related research in GUI
compatibility testing, we propose a benchmark of GUI compatibility testing. In
our experiments, we compare the replay success rate of existing tools. Based on
the experimental results, we summarize causes which may lead to ineffectiveness
in test case replay and propose opportunities for improving the
state-of-the-art