StateLens: A Reverse Engineering Solution for Making Existing Dynamic Touchscreens Accessible

Blind people frequently encounter inaccessible dynamic touchscreens in their everyday lives that are difficult, frustrating, and often impossible to use independently. Touchscreens are often the only way to control everything from coffee machines and payment terminals, to subway ticket machines and in-flight entertainment systems. Interacting with dynamic touchscreens is difficult non-visually because the visual user interfaces change, interactions often occur over multiple different screens, and it is easy to accidentally trigger interface actions while exploring the screen. To solve these problems, we introduce StateLens - a three-part reverse engineering solution that makes existing dynamic touchscreens accessible. First, StateLens reverse engineers the underlying state diagrams of existing interfaces using point-of-view videos found online or taken by users using a hybrid crowd-computer vision pipeline. Second, using the state diagrams, StateLens automatically generates conversational agents to guide blind users through specifying the tasks that the interface can perform, allowing the StateLens iOS application to provide interactive guidance and feedback so that blind users can access the interface. Finally, a set of 3D-printed accessories enable blind people to explore capacitive touchscreens without the risk of triggering accidental touches on the interface. Our technical evaluation shows that StateLens can accurately reconstruct interfaces from stationary, hand-held, and web videos; and, a user study of the complete system demonstrates that StateLens successfully enables blind users to access otherwise inaccessible dynamic touchscreens.Comment: ACM UIST 201

Adapt-First: a MDE Transformation Approach for Supporting User Interface Adaptation

Adapting user interfaces to different contexts of use is essential to enhance usability. Adaptation enhances user satisfaction by meeting changing context of use requirements. However, given the variety of contexts of use, and the significant amount of involved information and contextual treatments, transformations of user interface models that consider adaptation become complex. This complexity becomes a challenge when trying to add new adaptation rules or modify the transformation. In this paper, we present “Adapt-first”, an adaptation approach intended to simplify adaptation within model based user interfaces. It capitalizes on differentiating adaptations and concretization via two transformation techniques: concretization and translation. First-Adapt approach aims at reducing complexity and maintenance efforts of transformations from a model to another

Analysing Reverse Engineering Techniques for Interactive Systems

Reverse engineering is the process of discovering a model of a software system by analyzing its structure and functions. Reverse engineering techniques applied to interactive software applications (e.g. applications with user interfaces (UIs)) are very important and significant, as they can help engineers to detect defects in the software and then improve or complete them. There are several approaches, and many different tools, which are able to reverse-engineer software applications into formal models. These can be classified into two main types: dynamic tools and static tools. Dynamic tools interact with the application to find out the run-time behaviours of the software, simulating the actions of a user to explore the system’s state space, whereas static tools focus on static structure and architecture by analysing the code and documents. Reverse engineering techniques are not common for interactive software systems, but nowadays more and more organizations recognize the importance of interactive systems, as the trend in software used in computers is for applications with graphical user interfaces. This has in turn led to a developing interest in reverse engineering tools for such systems. Many reverse engineering tools generate very big models which make analysis slow and resource intensive. The reason for this is the large amount of information that is generated by the existing reverse engineering techniques. Slicing is one possible technique which helps with reducing un-necessary information for building models of software systems. This project focuses on static analysis and slicing, and considers how they can aid reverse engineering techniques for interactive systems, particularly with respect to the generation of a particular set of models, Presentation Models (PModels) and Presentation Interaction Models (PIMs)

Reverse engineering of GUI models

Tese de mestrado integrado. Engenharia Informática e Computação. Faculdade de Engenharia. Universidade do Porto. 200

Migration d'interfaces utilisateurs textuelles de systèmes patrimoniaux vers le Web

Les interfaces utilisateurs détiennent une importance non négligeable dans tout logiciel applicatif, car elles constituent le premier contact entre les utilisateurs et le logiciel. La migration d'interfaces utilisateurs textuelles de systèmes patrimoniaux vers le Web vise à remédier les problèmes liés à l'accès limité de tels systèmes tout en améliorant l'aspect et la convivialité de ces interfaces. Fréquemment, cela implique à la fois le changement de la plate-forme d'exécution - d'une plate-forme textuelle vers une architecture Web - et la refonte complète des interfaces vers une technologie de présentation Web. Ce mémoire est une synthèse des méthodes et des techniques de migration d'interfaces utilisateurs textuelles vers le Web. Nous y abordons, dans un premier temps, les notions reliées aux systèmes patrimoniaux et au Web. Cela permet d'établir le contexte de notre étude à savoir l'environnement de départ et l'environnement cible dans une migration d'interfaces utilisateurs vers le Web. Dans un deuxième temps, nous exposons différentes techniques et approches de migration d'interfaces utilisateurs. Finalement, nous terminons notre présentation avec trois exemples de plates-formes de migration d'interfaces utilisateurs

Reverse engineering of web applications

The MAP-i Doctoral Program of the Universities of Minho, Aveiro and PortoEven so many years after its genesis, the Internet is still growing. Not only are the users increasing, so are the number of different programming languages or frameworks for building Web applications. However, this plethora of technologies makes Web applications’ source code hard to comprehend and understand, thus deteriorating both their debugging and their maintenance costs. In this context, a number of proposals have been put forward to solve this problem. While, on one hand, there are techniques that analyze the entire source code of Web applications, the diversity of available implementation technology makes these techniques return unsatisfactory results. On the other hand, there are also techniques that dynamically (but blindly) explore the applications by running them and analyzing the results of randomly exploring them. In this case the results are better, but there is always the chance that some part of the application might be left unexplored. This thesis investigates if an hybrid approach combining static analysis and dynamic exploration of the user interface can provide better results. FREIA, a framework developed in the context of this thesis, is capable of analyzing Web applications automatically, deriving structural and behavioral interface models from them.Mesmo decorridos tantos anos desde a sua génese, a Internet continua a crescer. Este crescimento aplica-se não só ao número de utilizadores como também ao número de diferentes linguagens de programação e frameworks utilizadas para a construção de aplicações Web. No entanto, esta pletora de tecnologias leva a que o código fonte das aplicações Web seja difícil de compreender e analisar, deteriorando tanto o seu depuramento como os seus custos de manutenção. Neste contexto, foram desenvolvidas algumas propostas com intuito de resolver este problema. Não obstante, por um lado, existirem técnicas que analisam a totalidade do código fonte das aplicações Web, a diversidade das tecnologias de implementação existentes fazem com que estas técnicas gerem resultados insatisfatórios. Por outro lado, existem também técnicas que, dinamicamente (apesar de cegamente), exploram as aplicações, executando-as e analisando os resultados da sua exploração aleatória. Neste caso, os resultados são melhores, mas corremos o risco de ter deixado alguma parte da aplicação por explorar. Esta tese investiga se uma abordagem híbrida, combinando a análise estática com a exploração dinâmica da interface do utilizador consegue produzir melhores resultados. FREIA, uma framework desenvolvida no contexto desta tese é capaz de, automaticamente, analisar aplicações Web, derivando modelos estruturais e comportamentais da interface das mesmas.Esta investigação foi financiada pela Fundação para a Ciência e Tecnologia através da concessão de uma bolsa de doutoramento (SFRH/BD/71136/2010) no âmbito do Programa Operacional Potencial Humano (POPH), comparticipado pelo Fundo Social Europeu e por fundos nacionais do QREN

Reverse Engineering of User Interfaces

Reverse Engineering of User Interfaces(PhD in Management Sciences, option "Information Systems") -- UCL, 200

Reverse engineering of user interfaces : inference of structural and behavioural specifications

Many older systems, especially in the data processing domain, have a character oriented user interface. Reengineering these old interfaces to make them more user friendly would extend the life of the systems in which they are embedded.This thesis specifically addresses the issue of user interface reverse engineering. It attempts to define a method and to develop tools to assist programmers in obtaining structural and behavioural specifications of existing user interfaces. These specifications can be used to redesign new interfaces, generate new ones and integrate them into the old systems.A research prototype to infer user interface specifications from COBOL/CICS source code was developed. Using this tool, we successfully produced specifications of the user interface of an industrial MIS system