Generative AI as a New Assistive Technology for Web Interaction

Abstract

For users who are unfamiliar with technology or rely on assistive tools such as screen readers, interacting with a web page can be challenging. Ensuring a seamless experience requires a well-designed user interface (UI) that prioritizes accessibility and usability. However, achieving this target demands specialized expertise from developers and can involve significant effort. In this context, Generative Artificial Intelligence (GAI) has become a valuable aid for improving access to information and facilitating interaction with web interfaces. To effectively enhance user interaction---such as accessing services or specific functionalities---AI-driven tools must first be capable of understanding the structure and content of a web page. This study investigates if GAIs can be exploited to assist the user when navigating through a website, describing the site contents, explaining the interface structure and interactive elements, and suggesting actions or procedures to follow to perform a certain task or accomplish a specific goal. This kind of assistive technology can benefit not only visually impaired people but also persons with cognitive impairment and, more generally, people that are not ``skilled'' with modern web applications, like seniors. Specifically, thirteen popular websites were analyzed by asking Copilot one hundred questions. Results suggest that GAIs have the potential to assist people in web tasks. However, limitations have still been detected, with 20{\%} of completely erroneous answers received from the navigation and interaction questions and 15{\%} for those related to structure, mainly detected in pages having scarce accessibility and sites having a complex HTML structure, respectively