Automated visual classification of DOM-based presentation failure reports for responsive web pages

Since it is common for the users of a web page to access it through a wide variety of devices—including desktops, laptops, tablets and phones—web developers rely on responsive web design (RWD) principles and frameworks to create sites that are useful on all devices. A correctly implemented responsive web page adjusts its layout according to the viewport width of the device in use, thereby ensuring that its design suitably features the content. Since the use of complex RWD frameworks often leads to web pages with hard‐to‐detect responsive layout failures (RLFs), developers employ testing tools that generate reports of potential RLFs. Since testing tools for responsive web pages, like ReDeCheck, analyse a web page representation called the Document Object Model (DOM), they may inadvertently flag concerns that are not human visible, thereby requiring developers to manually confirm and classify each potential RLF as a true positive (TP), false positive (FP), or non‐observable issue (NOI)—a process that is time consuming and error prone. The conference version of this paper presented Viser, a tool that automatically classified three types of RLFs reported by ReDeCheck. Since Viser was not designed to automatically confirm and classify two types of RLFs that ReDeCheck's DOM‐based analysis could surface, this paper introduces Verve, a tool that automatically classifies all RLF types reported by ReDeCheck. Along with manipulating the opacity of HTML elements in a web page, as does Viser, the Verve tool also uses histogram‐based image comparison to classify RLFs in web pages. Incorporating both the 25 web pages used in prior experiments and 20 new pages not previously considered, this paper's empirical study reveals that Verve's classification of all five types of RLFs frequently agrees with classifications produced manually by humans. The experiments also reveal that Verve took on average about 4 s to classify any of the RLFs among the 469 reported by ReDeCheck. Since this paper demonstrates that classifying an RLF as a TP, FP, or NOI with Verve, a publicly available tool, is less subjective and error prone than the same manual process done by a human web developer, we argue that it is well‐suited for supporting the testing of complex responsive web pages

Automatically identifying potential regressions in the layout of responsive web pages

Providing a good user experience on the ever-increasing number and variety of devices being used to browse the web is a difficult, yet critical, task. With Responsive Web Design (RWD), front-end web developers design web pages so that they dynamically resize and rearrange content to best fit the dimensions of a device’s screen. However, when making code modifications to a responsive page, developers can easily introduce regressions from the correct layout that have detrimental effects at unpredictable screen sizes. For instance, the source code change that a developer makes to improve the layout at one screen size may obscure a page’s content at other sizes. Current approaches to testing are often insufficient because they rely on limited tools and error-prone manual inspections of a web page. As such, many unintended regressions in web page layout often go undetected and ultimately manifest in production web sites. To address the challenge of detecting regressions in responsive web pages, this paper presents an automated approach that extracts the responsive layout of two versions of a page and compares them, alerting developers to the differences in layout that they may wish to investigate further. We implemented the approach and empirically evaluated it on 15 real-world responsive web pages. Leveraging code mutations that a tool automatically injected into the pages as a systematic simulation of developer changes, the experiments show that the approach was highly effective. When compared with manual and automated baseline testing techniques, it detected 12.5% and 18.75% more injected changes, respectively. Along with identifying the best parameters for the method that extracts the responsive layout, the experiments show that the approach surpasses the baselines across changes that vary in their impact, but works particularly well for subtle, hard-to-detect mutants, showing the benefits of automatically identifying regressions in web page layout

Automatic Identification of Presentation Failures in Responsive Web Pages

With the increasing number and variety of devices being used to access the World Wide Web, providing a good browsing experience to all users, regardless of device, is a critical task. To do this, many web developers now use responsive web design (RWD) to build web pages that provide a bespoke layout tailored to the specific characteristics of the device in use, normally the viewport width. However, implementing responsive web pages is an error-prone task, as web page elements can behave in unpredictable ways as the viewport expands and contracts. This leads to presentation failures — errors in the visual appearance of the web page. As well-designed responsive web pages can have an array of benefits, identifying presentation failures quickly and accurately is an important task. Unfortunately, current approaches to detecting presentation failures in web pages are insufficient. The huge number of different viewport widths that require support makes thorough checking of the layout on all devices infeasible. Furthermore, the current range of developer tools only provide limited support for testing responsive web pages. This thesis tackles these problems by making the following contributions. First, it proposes the responsive layout graph (RLG), a model of the dynamic layout of modern responsive web pages. Then, it explores how the RLG can be used to automatically detect potentially unseen side-effects of small changes to the source code of a web page. Next, it investigates the detection of several common types of layout failures, leveraging implicit oracle information in place of an explicit oracle. Experiments showed both the approach for detecting potentially unseen side-effects and the approach for identifying common types of layout failure to be highly effective. The manual effort required by the user is further reduced by an approach that automatically grouped related failures together. Finally, a case study of 33 real-world responsive layout failures investigates how difficult such failures are to fix. These approaches have all been implemented into a software tool, ReDeCheck, which helps web developers create better responsive web pages

Testing web applications through layout constraints : tools and applications

With businesses trying to minimize cost, web applications are becoming increasingly useful by providing cross-platform applications that can be easily ported to desktop and mobile. However, testing these applications is much more arduous than testing desktop applications due to the lack of tools. In this thesis, we aim to describe a web application testing tool called Cornipickle which uses an expressive layout specification. The specification language for Cornipickle makes statements on applications’ layouts but we show that it can also test behaviours. With it, we effectively achieve to detect the bugs found with our survey of 35 real-world web applications. We also show Cornipickle’s versatility by using it in multiple applications and tools, notably pairing it with a crawler. Puisque les entreprises tentent de minimiser leurs coûts, les applications web deviennent de plus en plus utiles en fournissant des applications multiplateformes qui peuvent être facilement portées aux appareils mobiles et environnements de bureau. Par contre, tester ces applications est beaucoup plus ardu que de tester des applications de bureau en raison du manque d’outils. Dans ce mémoire, nous décrivons un outil de test d’applications web, Cornipickle, qui utilise une spécification expressive sur la mise en page. Le langage de spécification de Cornipickle fait des affirmations sur la mise en page des applications mais nous montrons qu’il permet aussi de tester des comportements. Grâce à lui, nous réussissons à détecter efficacement les bogues identifiés dans une étude de 35 applications webs. Nous montrons aussi la polyvalence de Cornipickle en l’utilisant dans une multitude d’applications et outils, notamment en l’associant à un robot d’indexation

Design, Integration, and Evaluation of IoT-Based Electrochromic Building Envelopes for Visual Comfort and Energy Efficiency

Electrochromic glazing has been identified as the next-generation high-performance glazing material for building envelopes due to its dynamic properties, which allow the buildings to respond to various climate conditions. IoT technologies have improved the sensing, communication, and interactions of building environmental data. Few studies have been done to synthesize the advancements in EC materials and building IoT technologies for better building performance. The challenge remains in the lack of compatible design and simulation tools, limited understanding of integration, and a paucity of evaluation measures to support the convergence between the EC building envelopes and IoT technologies. This research first explores the existing challenges of using EC building envelopes using secondary data analysis and case studies. An IoT-based EC prototype system is developed to demonstrate the feasibility of IoT and EC integration. Functionalities, reliability, interoperability, and scalability are assessed with comparisons of four alternative building envelope systems. Nation-wide evaluations of EC building performance are conducted to show regional differences and trade-offs of visual comfort and energy efficiency. A machine learning approach is proposed to solve the predictive EC control problem under random weather conditions. The best prediction models achieve 91.08% mean accuracy with the 16-climate-zone data set. The importance of predictive variables is also measured in each climate zone to develop a better understanding of the effectiveness of climatic sensors. Additionally, a simulation study is conducted to investigate the relationships between design factors and EC building performance. An instantaneous daylight measure is developed to support active daylight control with IoT-based EC building envelopes

Hypatiamat - I want to solve questions about...

Dissertação de mestrado em Informatics EngineeringHypatiamat is a Portuguese project comprised of several applications that aim to develop the Math skills of students from the 1st through 9th grades (Basic Education). The ingraining of mental calculation strategies, numbering systems, and logical operations lead to a better success rate in this subject in later years. One of the project’s components is the online platform (https://www.hypatiamat.com), which aims to foster autonomous learning through more interactive practices due to the current ease of technological access in this age group, by trying to appropriate teaching to everyday life. Several tools are made available, such as videos, tutorials, explanations, questions, etc. on various Math topics that students can easily access at any time. Teachers that aim to enhance their students’ learning process using this digital approach can exercise it in multiple applications provided by the platform, where the interactions are carried out and controlled through these means. The monolithic architecture (written in PHP) has received contributions from multiple developers over the years in order to address the scalability issues introduced with this platform’s growing popularity, which thus far demanded manual efforts for maintenance and content insertion. As such, there has been an incremental process of modernization, turning the various constituent applications into distinct microservices. "I Want to Solve Questions About..." is one of these applications where students are provided with a large selection of questions in the form of mini-games (multiple choice, true or false, ...), regarding the themes mentioned above. The first objective of the dissertation is to develop a back-office that allows the teachers in charge of the project to manage existing questions as well as add new ones for the students, since the current process requires updating the database manually. The second one is the modernization of the application’s interface at the technological level, by making use of adequate frameworks and programming languages and at the user level, by making an effort to maintain the intuitive workflow that led to its popularity but with a modernized design, in order to be consistent with other online tools.O Hypatiamat é um projeto português constituído por várias aplicações que visa desenvolver as aptidões, na disciplina de Matemática, de alunos do 1º ao 9º ano de escolaridade (Educação Básica). O enraizamento de estratégias de cálculo mental, sistemas de numeração e operações lógicas originam uma melhor taxa de sucesso nesta disciplina em anos posteriores. Uma das componentes deste projeto é a plataforma online (https://www.hypatiamat.com), cujo propósito é fomentar a aprendizagem autónoma através de práticas mais interativas, devido à facilidade de acesso tecnológico atual desta faixa etária, tentando apropriar o ensino ao quotidiano. São disponibilizadas várias ferramentas, tais como vídeos, tutoriais, explicações, questões, etc sobre os vários temas da Matemática (Ensino Básico) que os alunos podem facilmente aceder a qualquer momento. Professores que pretendam enriquecer a aprendizagem dos seus alunos com esta metodologia digital podem exercê-lo nas várias aplicações que a plataforma disponibiliza, onde a interação é realizada e controlada através destes meios. A arquitetura monolítica (escrita em PHP) tem recebido contribuições de vários desenvolvedores ao longo dos anos de modo a colmatar os problemas de escalabilidade introduzidos com a popularidade crescente desta plataforma, que até agora exigia esforço manual para manutenção e inserção de conteúdo. Assim, tem existido um processo incremental de modernização, tornando as várias aplicações constituintes em microsserviços distintos. A "Quero resolver questões de..." é uma destas aplicações, onde são disponibilizadas aos alunos várias questões, sob a forma de mini-jogos (escolha múltipla, verdadeiro ou falso, ...), relativas aos temas mencionados anteriormente. O primeiro objetivo da dissertação é o desenvolvimento de um backoffice que permita aos professores responsáveis gerirem as questões existentes assim como adicionarem novas para os alunos, visto que o processo atual obriga a atualização manual na base de dados. O segundo é a modernização da interface da aplicação ao nível: tecnológico, utilizando frameworks e linguagens de programação adequadas ao problema; do utilizador, de modo a manter o fluxo intuitivo que gerou a sua popularidade mas tendo em conta um design mais atualizado para manter a consistência com outras ferramentas online

Enabling Collaborative Visual Analysis across Heterogeneous Devices

We are surrounded by novel device technologies emerging at an unprecedented pace. These devices are heterogeneous in nature: in large and small sizes with many input and sensing mechanisms. When many such devices are used by multiple users with a shared goal, they form a heterogeneous device ecosystem. A device ecosystem has great potential in data science to act as a natural medium for multiple analysts to make sense of data using visualization. It is essential as today's big data problems require more than a single mind or a single machine to solve them. Towards this vision, I introduce the concept of collaborative, cross-device visual analytics (C2-VA) and outline a reference model to develop user interfaces for C2-VA. This dissertation covers interaction models, coordination techniques, and software platforms to enable full stack support for C2-VA. Firstly, we connected devices to form an ecosystem using software primitives introduced in the early frameworks from this dissertation. To work in a device ecosystem, we designed multi-user interaction for visual analysis in front of large displays by finding a balance between proxemics and mid-air gestures. Extending these techniques, we considered the roles of different devices–large and small–to present a conceptual framework for utilizing multiple devices for visual analytics. When applying this framework, findings from a user study showcase flexibility in the analytic workflow and potential for generation of complex insights in device ecosystems. Beyond this, we supported coordination between multiple users in a device ecosystem by depicting the presence, attention, and data coverage of each analyst within a group. Building on these parts of the C2-VA stack, the culmination of this dissertation is a platform called Vistrates. This platform introduces a component model for modular creation of user interfaces that work across multiple devices and users. A component is an analytical primitive–a data processing method, a visualization, or an interaction technique–that is reusable, composable, and extensible. Together, components can support a complex analytical activity. On top of the component model, the support for collaboration and device ecosystems comes for granted in Vistrates. Overall, this enables the exploration of new research ideas within C2-VA