GUI Element Identification with Semantic Mapping

User Interface test automation faces significant obstacles due to test failures connected to application changes. Additionally, current User Interface testing methods are not context aware and usage-based, which makes exploring web application functionality challenging. Robots used for crawling web application interfaces are slow and do not reflect human interaction with them. Semantic mapping (semantic matching) has been proposed as a method for reusing existing tests between web applications in the same domain to mitigate issues with testing speed and context awareness. This thesis explores semantic mapping for robust User Interface element identification that could alleviate the issue with test failures upon application changes. Semantic mapping uses textual cues of User Interface elements neighboring testable features to identify similar features in other applications of the same domain. This work argues that the same technique can be applied to various versions of the same web application. Existing tools leverage text attributes of features' neighbors based on the hierarchy and position of an element, while this study applies semi-supervised learning methods to extract relevant text from elements surrounding features. It uses state-of-the-art pre-trained language models for embedding textual cues. To find similar features, it uses cosine similarity between sentences as a measure of semantic similarity. This implementation of semantic matching has demonstrated promising results for User Interface element identification between two versions of the same web application

AI2D-RST : A multimodal corpus of 1000 primary school science diagrams

This article introduces AI2D-RST, a multimodal corpus of 1000 English-language diagrams that represent topics in primary school natural sciences, such as food webs, life cycles, moon phases and human physiology. The corpus is based on the Allen Institute for Artificial Intelligence Diagrams (AI2D) dataset, a collection of diagrams with crowdsourced descriptions, which was originally developed to support research on automatic diagram understanding and visual question answering. Building on the segmentation of diagram layouts in AI2D, the AI2D-RST corpus presents a new multi-layer annotation schema that provides a rich description of their multimodal structure. Annotated by trained experts, the layers describe (1) the grouping of diagram elements into perceptual units, (2) the connections set up by diagrammatic elements such as arrows and lines, and (3) the discourse relations between diagram elements, which are described using Rhetorical Structure Theory (RST). Each annotation layer in AI2D-RST is represented using a graph. The corpus is freely available for research and teaching.Peer reviewe

Designing for Web Usability and Accessibility : User-Interface Design Guidelines in Connection with Human-Computer Interaction

This thesis reports on the user-interface design guidelines for usability and accessibility in their connection to human-computer interaction and their implementation in the web design. The goal is to study the theoretical background of the design rules and apply them in designing a real-world website. The analysis of Jakobson’s communication theory applied in the web design and its implications in the design guidelines of visibility, affordance, feedback, simplicity, structure, consistency and tolerance is conducted in order to shape the criteria used in accomplishing the practical part of the study. It concludes creation of the website design according to the design rules. The project has been successfully conducted, and the design stage of the website development has been completed with an aim to enhance the degree of usability and accessibility of the product through the design. The results of the analysis and the website design can be used for further investigation of the user-interface design guidelines. In addition, they can be implemented in other real-world websites

Wassup with the language of Twitter : an exploratory analysis of structural features

Twitter has become a staple social media platform for millions of English speakers of different socioeconomic categories, which makes it a valuable source of research material for linguistic analysis. There has not been many quantitative studies of structural language features in digital communication that use a representative sample of online users, which is why this investigation attempts to contribute to the body of research with an exploratory analysis of a Twitter corpus. Researchers have previously identified numerous linguistic phenomena, for example, creative abbreviations, non-standard capitalization, punctuation and spelling, which appear with varying frequencies in different types of computer-mediated communication. Although such features are not limited to digital platforms, they often function as expressivity devices and contribute to the speechlike quality of written language in text-based electronic media. This study presents a quantitative analysis of common typographic and orthographic characteristics of the language of Twitter with the goal to identify the reasons for their popularity. Since this work also seeks to evaluate Twitter from the point of view of spoken and written discourses, the investigation additionally focuses on the lexical density of tweets and the distributions of parts-of-speech and personal pronouns. The findings are compared to previous research results that were obtained from studying instant and text messaging, emails and discussion forums. The analysis of the language of Twitter suggests that the use of characteristic formal features is frequently dictated by the need to express emotions and emulate nonverbal cues in the communication environment lacking sound. Additionally, certain cases of nonstandard typography and orthography are caused by the character limitation of the platform and users trying to minimize the number of keystrokes. The distributions of major word classes and personal pronouns show that Twitter, unlike instant and text messaging, serves other purposes besides personal communication, for example, news reporting, social networking, political campaigning and advertising, in this sense, it is similar to emails and electronic forums