Reverseorc:Reverse engineering of resizable user interface layouts with or-constraints

Reverse engineering (RE) of user interfaces (UIs) plays an important role in software evolution. However, the large diversity of UI technologies and the need for UIs to be resizable make this challenging. We propose ReverseORC, a novel RE approach able to discover diverse layout types and their dynamic resizing behaviours independently of their implementation, and to specify them by using OR constraints. Unlike previous RE approaches, ReverseORC infers flexible layout constraint specifications by sampling UIs at different sizes and analyzing the differences between them. It can create specifications that replicate even some non-standard layout managers with complex dynamic layout behaviours. We demonstrate that ReverseORC works across different platforms with very different layout approaches, e.g., for GUIs as well as for the Web. Furthermore, it can be used to detect and fix problems in legacy UIs, extend UIs with enhanced layout behaviours, and support the creation of flexible UI layouts.Comment: CHI2021 Full Pape

Unified GUI adaptation in Dynamic Software Product Lines

In the modern world of mobile computing and ubiquitous technology, society is able to interact with technology in new and fascinating ways. To help provide an improved user experience, mobile software should be able to adapt itself to suit the user. By monitoring context information based on the environment and user, the application can better meet the dynamic requirements of the user. Similarly, it is noticeable that programs can require different static changes to suit static requirements. This program commonality and variability can benefit from the use of Software Product Line Engineering, reusing artefacts over a set of similar programs, called a Software Product Line (SPL). Historically, SPLs are limited to handling static compile time adaptations. Dynamic Software Product Lines (DSPL) however, allow for the program configuration to change at runtime, allow for compile time and runtime adaptation to be developed in a single unified approach. While currently DSPLs provide methods for dealing with program logic adaptations, variability in the Graphical User Interface (GUI) has largely been neglected. Due to this, depending on the intended time to apply GUI adaptation, different approaches are required. The main goal of this work is to extend a unified representation of variability to the GUI, whereby GUI adaptation can be applied at compile time and at runtime. In this thesis, an approach to handling GUI adaptation within DSPLs, providing a unified representation of GUI variability is presented. The approach is based on Feature-Oriented Programming (FOP), enabling developers to implement GUI adaptation along with program logic in feature modules. This approach is applied to Document-Oriented GUIs, also known as GUI description languages. In addition to GUI unification, we present an approach to unifying context and feature modelling, and handling context dynamically at runtime, as features of the DSPL. This unification can allow for more dynamic and self-aware context acquisition. To validate our approach, we implemented tool support and middleware prototypes. These different artefacts are then tested using a combination of scenarios and scalability tests. This combination first helps demonstrate the versatility and its relevance of the different approach aspects. It further brings insight into how the approach scales with DSPL size

The web as a runtime in mobile context

Web-teknologiat kehitettiin alun perin kuvaamaan staattisten web-sivujen sisältöä. Web-selainten suosion vuoksi samoja teknologioita hyödynnetään nykyisin myös sovellusten toteuttamiseen käyttäen web-selainta niiden suorittamiseen vuorovaikutteisesti. Web-teknologioiden suosiosta huolimatta ne sisältävät useita ongelmia ohjelmistojen toteuttamisen näkökulmasta. Lisäksi mobiililaitteiden rajoitukset tekevät kaikilla laitteilla toimivien sovellusten toteuttamisesta haasteellista. Tämän vuoksi uusia ohjelmistoalustoja on kehitetty ratkaisemaan web-selainten asettamia rajoituksia. Tässä diplomityössä koostetaan vaatimukset mobiililaitteissa toimiville web-sovelluksille. Tämän lisäksi web-sovelluksille luodaan luokitusjärjestelmä ja tärkeimmät web-sovelluskehitykseen liittyvät web-teknologiat ja ohjelmistoalustat esitellään. Esimerkkisovellus toteutetaan käyttäen web-teknologioita ja hyödyntäen ohjelmistoalustan ominaisuuksia. Lisäksi esimerkkisovellus arvioidaan määritettyä vaatimusmäärittelyä vastaan. Arviointi paljasti lukuisia haasteita, jotka liittyivät web-teknologioiden puutteelliseen ilmaisuvoimaan, yhteentoimivuuteen ja ohjelmistoalustan toiminnallisuuteen. Tästä huolimatta esimerkkisovellus toteutti sille asetetut vaatimukset työpöytäohjelmistotasoisesta toiminnallisuudesta.Web technologies were initially designed to facilitate the creation of static web pages. However, the ubiquity of the web browser has motivated the use of the same technologies as a basis for desktop-style applications which are executed within the web browser and have their characteristics such as high interactivity. Despite the popularity of web applications, there exists various problems due to the fact that established web technologies were not specified with applications in mind. In addition, the constraints introduced by mobile devices challenge the ubiquity of such applications. On this account, new platforms have emerged extending the capabilities of web browsers. In this thesis, the requirements for client-side web applications in mobile context are synthesized. Moreover, a taxonomy for web applications is drawn and client-side web technologies and major software platforms relevant to the client-side web applications are discussed. Furthermore, an application concept implementation developed using web technologies leveraging the capabilities of the major mobile platform is presented and evaluated against the defined requirements. The evaluation revealed various problems related to limited expressiveness of web technologies, interoperability and platform functionality. Regardless, the implementation provided a level of functionality comparable to that of native applications

Case Study: Porting Qt to Windows Runtime

With the abundance of operating system choices available to end-users, particularly for mobile devices, application developers look for ways to cut development time while increasing the portability and maintainability of their source code. One solution to this challenge can be found through use of cross-platform frameworks. Cross-platform frameworks function by abstracting the system-specific details of incompatible platforms into a common programming interface which developers can use to target many different devices and operating systems. This thesis studies the abstraction architecture of Qt, a leading cross-platform C++ graphical user interface framework, with the goal of bringing a new platform, Windows Runtime, to the framework's set of supported targets. Windows Runtime is a collective programming interface for the Microsoft Windows 8 family of operating systems, including Windows 8, Windows Phone 8, and Windows RT. While Qt already supports a range of desktop and mobile operating systems -- including Windows, Mac OSX, Linux/X11, Android, iOS, BlackBerry, and Sailfish -- support for Windows Runtime is a new feature of the framework brought forth by this case study. Current trends in cross-platform frameworks, particularly declarative user interface frameworks with a mobile emphasis, are assessed and compared to Qt's offering, and the implementation of Qt for Windows Runtime is prepared with these trends in mind. The implementation contributes to the open-source Qt Project, with the contributions included in the official Qt 5.3 release. Using the released version of Qt 5.3, a canonical Qt application is ported to the new platform and is certified and published in the Windows Store. Through this porting and publication process, an evaluation of the project's success is constructed within a cross-platform context. The outlook for Windows Runtime as a growing platform is positive, as is the outlook for the uptake of Qt (and cross-platform frameworks in general) within modern device ecosystems. Moving forward, the quality and feature parity of Qt for Windows Runtime (as compared to competing frameworks) is expected to improve as users and open-source contributors make this new offering part of their respective development workflows and software projects

Bridging the Gap Between People, Mobile Devices, and the Physical World

Human-computer interaction (HCI) is being revolutionized by computational design and artificial intelligence. As the diversity of user interfaces shifts from personal desktops to mobile and wearable devices, yesterday’s tools and interfaces are insufficient to meet the demands of tomorrow’s devices. This dissertation describes my research on leveraging different physical channels (e.g., vibration, light, capacitance) to enable novel interaction opportunities. We first introduce FontCode, an information embedding technique for text documents. Given a text document with specific fonts, our method can embed user-specified information (e.g., URLs, meta data, etc) in the text by perturbing the glyphs of text characters while preserving the text content. The embedded information can later be retrieved using a smartphone in real time. Then, we present Vidgets, a family of mechanical widgets, specifically push buttons and rotary knobs that augment mobile devices with tangible user interfaces. When these widgets are attached to a mobile device and a user interacts with them, the nonlinear mechanical response of the widgets shifts the device slightly and quickly. Subsequently, this subtle motion can be detected by the Inertial Measurement Units (IMUs), which is commonly installed on mobile devices. Next, we propose BackTrack, a trackpad placed on the back of a smartphone to track finegrained finger motions. Our system has a small form factor, with all the circuits encapsulated in a thin layer attached to a phone case. It can be used with any off-the-shelf smartphone, requiring no power supply or modification of the operating systems. BackTrack simply extends the finger tracking area of the front screen, without interrupting the use of the front screen. Lastly, we demonstrate MoiréBoard, a new camera tracking method that leverages a seemingly irrelevant visual phenomenon, the moiré effect. Based on a systematic analysis of the moiré effect under camera projection, MoiréBoard requires no power nor camera calibration. It can easily be made at a low cost (e.g., through 3D printing) and ready to use with any stock mobile device with a camera. Its tracking algorithm is computationally efficient and can run at a high frame rate. It is not only simple to implement, but also tracks devices at a high accuracy, comparable to the state-of-the-art commercial VR tracking systems

A Model-Based AI-Driven Test Generation System

Achieving high software quality today involves manual analysis, test planning, documentation of testing strategy and test cases, and development of automated test scripts to support regression testing. This thesis is motivated by the opportunity to bridge the gap between current test automation and true test automation by investigating learning-based solutions to software testing. We present an approach that combines a trainable web component classifier, a test case description language, and a trainable test generation and execution system that can learn to generate new test cases. Training data was collected and hand-labeled across 7 systems, 95 web pages, and 17,360 elements. A total of 250 test flows were also manually hand-crafted for training purposes. Various machine learning algorithms were evaluated. Results showed that Random Forest classifiers performed well on several web component classification problems. In addition, Long Short-Term Memory neural networks were able to model and generate new valid test flows

A WebGL application based on BIM IFC

Dissertação apresentada à Universidade Fernando Pessoa como partes dos requisitos para a obtenção do grau de Mestre em Engenharia Informática, ramo de Computação MóvelThe possibility of displaying high performance 3D accelerated graphics in the browser is seen as an obstacle to the conversion of applications to the web. The release of WebGL made Web3D gain new strength to overcome that obstacle. Architecture, Engineering and Construction (AEC) tools are a type of applications that could benefit with this advance. In the AEC industry, there is a standard candidate for Building Information Modelling (BIM), called Industry Foundation Classes (IFC). This data model promotes interoperability between AEC tools, giving a common format to the applications. This work comes from the need of redesigning a legacy application that allows the user to design, display and calculate building structures. Focusing on the displaying of building structures, this work merges IFC and WebGL into an application, to replicate in a modern way the legacy application capabilities. This is done by developing a server module that processes the IFC data model and a client module that displays that model in a WebGL environment. The result is a prototype web application capable of displaying 3D IFC building models in the browser without plug-ins. A possibilidade de visualização de gráficos acelerados 3D de alto desempenho no navegador ainda é visto como um obstáculo na migração de aplicações para a web. O lançamento do WebGL fez o Web3D ganhar uma nova força para superar esse obstáculo. As ferramentas de Arquitetura, Engenharia e Construção (AEC) são um tipo de aplicações que podem beneficiar com este avanço. Na indústria AEC, há um candidato a padrão para Building Information Modelling (BIM), chamado de Industry Foundation Classes (IFC). Este modelo de dados promove a interoperabilidade entre as ferramentas de AEC, fornecendo um formato comum às aplicações. Este trabalho surge da necessidade de redesenhar uma aplicação legada que permite ao o utilizador projetar, visualizar e calcular estruturas de edifícios. Focando na visualização de estruturas de edifícios, este trabalho funde o IFC e o WebGL numa aplicação, para replicar de forma moderna as capacidades da aplicação legada. Isto é feito através do desenvolvimento de um módulo de servidor que processa o modelo de dados IFC e um módulo de cliente que mostra esse modelo num ambiente WebGL. O resultado é um protótipo duma aplicação web capaz de visualizar modelos 3D de edifícios em formato IFC no browser sem plug-ins

Guiding Random Graphical and Natural User Interface Testing Through Domain Knowledge

Users have access to a diverse set of interfaces that can be used to interact with software. Tools exist for automatically generating test data for an application, but the data required by each user interface is complex. Generating realistic data similar to that of a user is difficult. The environment which an application is running inside may also limit the data available, or updates to an operating system can break support for tools that generate test data. Consequently, applications exist for which there are no automated methods of generating test data similar to that which a user would provide through real usage of a user interface. With no automated method of generating data, the cost of testing increases and there is an increased chance of bugs being released into production code. In this thesis, we investigate techniques which aim to mimic users, observing how stored user interactions can be split to generate data targeted at specific states of an application, or to generate different subareas of the data structure provided by a user interface. To reduce the cost of gathering and labelling graphical user interface data, we look at generating randomised screen shots of applications, which can be automatically labelled and used in the training stage of a machine learning model. These trained models could guide a randomised approach at generating tests, achieving a significantly higher branch coverage than an unguided random approach. However, for natural user interfaces, which allow interaction through body tracking, we could not learn such a model through generated data. We find that models derived from real user data can generate tests with a significantly higher branch coverage than a purely random tester for both natural and graphical user interfaces. Our approaches use no feedback from an application during test generation. Consequently, the models are “generating data in the dark”. Despite this, these models can still generate tests with a higher coverage than random testing, but there may be a benefit to inferring the current state of an application and using this to guide data generation

Recognizing and understanding user behaviors from screencasts

User interacts with computers or mobile devices, leading to user behaviors on screen. In the context of software engineering, analyzing user behavior enables many applications such as intelligent bug fix, code completion and knowledge recommendation for developers. Such technique can be extended to more general knowledge worker environment, in which users have to manipulate devices according to specific guidelines. Existing works rely heavily on software instrumentation to obtain user actions from operation systems, which is hard to deploy and maintain. In addition, considering the security and privacy of some scenarios, non-intrusive is the major requirement to be included in the system. In this work, we leverage Computer Vision and Natural Language Processing techniques to recognize and understand user behaviors from screencasts, which is a non-intrusive and cross-platform method. We first recognize 10 categories of low level user actions such as mouse moving and type text, then summarize them to higher level abstractions (i.e. line-granularity coding steps). We also try to interpret user interaction with applications by multi-task learning and generate structured language descriptions (i.e. command, widget and location). Finally, unsupervised learning method is introduced for GUI linting problem, which is taken as a case study of user behavior analysis. To train the deep neural networks, we collect diverse video data from YouTube, Twitch and Bugzilla, and manually label them to build the dataset. The experiment results demonstrate the high performance of proposed method, and the user study validate the practical applications of many downstream tasks