VisionAPI: An API for Offline and Online Segmentation and Identification of Hand-Sketched Graphical User Interfaces

Abstract

Segmentation and identification of a graphical user interface consist of detecting the location, dimensions, and arrangement of elements of the user interface, such as controls, labels, images, and icons, and recognizing them, respectively. While these problems have been already addressed for a graphical user interface stored in a file and processed offline, it has received less attention for online processing when the interface evolves and is expressed in different formats, such as a whiteboard drawing or a paper sketch. To overcome these limitations, we present VisionAPI, an application programming interface trained for segmenting and identifying elements of a hand-sketched graphical user interface both offline and online using computer vision. For this purpose, we rely on a software architecture based on Resnet101 to extract features and Faster R-CNN to build boundary boxes to obtain an 85% recognition rate for 21 classes of elements found in graphical user interfaces: paragraph, dropdown list, checkbox, radio button, rating, toggle button, text area, date picker, stepper input, slider, video, label, table, list, header, button, image, linebreak, container, link, and text input