1 research outputs found
Owl Eyes: Spotting UI Display Issues via Visual Understanding
Graphical User Interface (GUI) provides a visual bridge between a software
application and end users, through which they can interact with each other.
With the development of technology and aesthetics, the visual effects of the
GUI are more and more attracting. However, such GUI complexity posts a great
challenge to the GUI implementation. According to our pilot study of
crowdtesting bug reports, display issues such as text overlap, blurred screen,
missing image always occur during GUI rendering on different devices due to the
software or hardware compatibility. They negatively influence the app
usability, resulting in poor user experience. To detect these issues, we
propose a novel approach, OwlEye, based on deep learning for modelling visual
information of the GUI screenshot. Therefore, OwlEye can detect GUIs with
display issues and also locate the detailed region of the issue in the given
GUI for guiding developers to fix the bug. We manually construct a large-scale
labelled dataset with 4,470 GUI screenshots with UI display issues and develop
a heuristics-based data augmentation method for boosting the performance of our
OwlEye. The evaluation demonstrates that our OwlEye can achieve 85% precision
and 84% recall in detecting UI display issues, and 90% accuracy in localizing
these issues. We also evaluate OwlEye with popular Android apps on Google Play
and F-droid, and successfully uncover 57 previously-undetected UI display
issues with 26 of them being confirmed or fixed so far.Comment: Accepted to 35th IEEE/ACM International Conference on Automated
Software Engineering (ASE 20