Scripted GUI Testing of Android Apps: A Study on Diffusion, Evolution and Fragility

Background. Evidence suggests that mobile applications are not thoroughly tested as their desktop counterparts. In particular GUI testing is generally limited. Like web-based applications, mobile apps suffer from GUI test fragility, i.e. GUI test classes failing due to minor modifications in the GUI, without the application functionalities being altered. Aims. The objective of our study is to examine the diffusion of GUI testing on Android, and the amount of changes required to keep test classes up to date, and in particular the changes due to GUI test fragility. We define metrics to characterize the modifications and evolution of test classes and test methods, and proxies to estimate fragility-induced changes. Method. To perform our experiments, we selected six widely used open-source tools for scripted GUI testing of mobile applications previously described in the literature. We have mined the repositories on GitHub that used those tools, and computed our set of metrics. Results. We found that none of the considered GUI testing frameworks achieved a major diffusion among the open-source Android projects available on GitHub. For projects with GUI tests, we found that test suites have to be modified often, specifically 5\%-10\% of developers' modified LOCs belong to tests, and that a relevant portion (60\% on average) of such modifications are induced by fragility. Conclusions. Fragility of GUI test classes constitute a relevant concern, possibly being an obstacle for developers to adopt automated scripted GUI tests. This first evaluation and measure of fragility of Android scripted GUI testing can constitute a benchmark for developers, and the basis for the definition of a taxonomy of fragility causes, and actionable guidelines to mitigate the issue.Comment: PROMISE'17 Conference, Best Paper Awar

SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

Graphical User Interface (GUI) agents are designed to automate complex tasks on digital devices, such as smartphones and desktops. Most existing GUI agents interact with the environment through extracted structured data, which can be notably lengthy (e.g., HTML) and occasionally inaccessible (e.g., on desktops). To alleviate this issue, we propose a novel visual GUI agent -- SeeClick, which only relies on screenshots for task automation. In our preliminary study, we have discovered a key challenge in developing visual GUI agents: GUI grounding -- the capacity to accurately locate screen elements based on instructions. To tackle this challenge, we propose to enhance SeeClick with GUI grounding pre-training and devise a method to automate the curation of GUI grounding data. Along with the efforts above, we have also created ScreenSpot, the first realistic GUI grounding benchmark that encompasses mobile, desktop, and web environments. After pre-training, SeeClick demonstrates significant improvement in ScreenSpot over various baselines. Moreover, comprehensive evaluations on three widely used benchmarks consistently support our finding that advancements in GUI grounding directly correlate with enhanced performance in downstream GUI agent tasks. The model, data and code are available at https://github.com/njucckevin/SeeClick

Applying Scriptless Test Automation on Web Applications from the Financial Sector

[EN] This industry showcase presents experiences on application of testar, an open source tool for scriptless testing through graphical user interface (GUI), to the web applications of Kuveyt T¿urk Participation Bank in Turkey. Kuveyt T¿urk Bank uses Selenium and Appium for regression testing of mobile and internet banking, but the maintenance cost of the test scripts is increasing day by day. Therefore, scriptless GUI testing with testar was evaluated. To provide better support for testing web-based applications, testar was extended with Selenium WebDriver integration, JavaScript support, and other new features. Results show that testar detects GUI elements much better after the improvements, and it was able to find 2 relevant errors that were not identified by existing scripted test cases.This work has been partially funded by ITEA3 TESTOMAT Project, ITEA3 IVVES project7 and EU H2020 DECODER projecAho, P.; Buijs, G.; Akin, A.; Senturk, S.; Pastor-Ricós, F.; De Gouw, S.; Vos, TE. (2021). Applying Scriptless Test Automation on Web Applications from the Financial Sector. SISTEDES. 1-4. http://hdl.handle.net/10251/178254S1

Web-based monitoring of an automated fertigation system: An IoT application

Internet of Things (IoT) is an emerging system that incorporates many technologies from different areas. In this paper, we present the implementation of IoT in an agriculture industry, particularly in monitoring an automated fertigation system. The monitoring system comprises a web-based system, an automatic fertigation system and a communication network. The main focus of this paper is on the web-based system where the data from the SQLite database is used in the web-GUI to display parameters such as the status of water level, the flow condition of valves and pipes as well as the overall operation of automated fertigation system. The paper also described on how farmers can access the website, set fertigation schedule and determine fertilizer’s formulation. Different from others, this system is equipped with emergency mode to stop the fertigation system which can be controlled directly from the website. Our method uses a microprocessor to handle the databases, web-GUI and control communications between the fertigation system and the web-based system. This system will ease farmers in managing their automated fertigation system virtually using their mobile device

A web-based teleoperative mobile robotic system : Master of Engineering in Information Engineering at Massey University, Albany, Auckland, New Zealand

With the rapid development of internet technology, it becomes real that human beings can access, modify and control a remote hardware device via internet connection. Such remote operations can replace the human to be present at a dangerous or unreachable place or can make as many as possible users to access the hardware in different places at a low cost. The thesis research was aimed at developing a web based mobile robot control framework for education purpose. It should be composed of a mobile robot. Http server, dynamic user interface and video server. With it users can view and control the real robot via a normal web browser and can choose to run either simulation or the real robot. This is done by setting up operational parameters via a friendly GUI (graphic user interface). Users also can upload and compile their own C code to control the robot and get back the running results. The main objectives of this thesis research are hardware upgrading for Nomadic Super Scout mobile robot and web based php programming. For the first objective, the onboard PC was replaced by a laptop that is remotely placed and connected to the robot control system via Bluetooth wireless. The Nserver for robot simulation was set up in the Linux operating environment. For the second objective, the software programming was focused on building a web control platform which should be user friendly. An Apache server was developed where PHP program was used for the user interface. The main advantage of using PHP is that it does not need to install or download any software or script to get access to the remote robot via a normal web browser on any operation like windows or Linux. The web-based mobile robot system was tested using two different cases. One case demonstrated how the user specifies a set of motion parameters of the robot that is programmed to perform a wall-following behaviour. The other demonstrated how the user uploads a collision avoidance program to run the robot that is placed among obstacles. Both case studies were performed in real environments and the results proved the success of the developed web-based robotic system

Energy Measurement of SPDY Protocol on Mobile Platform

The past few years have witnessed an explosive growth in mobile Internet data traffic with web browsing being one of the key activities on mobile devices. There is tremendous interest in optimizing mobile web. In this regard, a new protocol called SPDY was introduced by Google to augment web browsing, however it\u27s impact on the device energy consumption is not clearly understood. In this work we evaluate the energy characteristics of SPDY-based web browsing on mobile devices. In order to measure the energy consumption of web activities, we use AT&T’s ARO [1]. This tool is widely accepted and used by the industry as well as academia for radio (LTE) energy measurement. However the tool was initially designed as a GUI and hence the efficiency for handling large-scaled data was compromised. The first part of the project involves optimizing ARO so that the program runs automatically on a fairly large amount of data with minimum time. In this process we reduced the running time of ARO by a factor of 13 in average compared to the baseline GUI version. In the second part of the project, we set up a SPDY proxy to be used in our evaluations, since there are limited number of websites that support SPDY at this moment. Further we conduct an initial evaluation comparing SPDY proxy-based download to traditional HTTP-based download. Our initial analysis shows that for 18 out of 20 pages that we evaluated SPDY is doing better than the existing HTTP energy wise. E.g. for 50% of pages, SPDY reduces the LTE energy consumption by at least 3J (\u3e20%) while for few pages the benefits are small and sometimes it is doing worse than HTTP. As part of future work, we will extend our evaluation to get a comprehensive understanding on the energy characteristics of SPDY and compare it to other better approaches

Pattern based GUI testing for mobile applications

This paper presents a study aiming to assess the feasibility of using the Pattern Based GUI Testing approach, PBGT, to test mobile applications. PBGT is a new model based testing approach that aims to increase systematization, reusability and diminish the effort in modelling and testing. It is based on the concept of User Interface Test Patterns (UITP) that contain generic test strategies for testing common recurrent behaviour, the so-called UI Patterns, on GUIs through its possible different implementations after a configuration step. Although PBGT was developed having web applications in mind, it is possible to develop drivers for other platforms in order to test a wide set of applications. However, web and mobile applications are different and only the development of a new driver to execute test cases over mobile applications may not be enough. This paper describes a study aiming to identify the adaptations and updates the PBGT should undergo in order to test mobile applications. © 2014 IEEE

Towards Assistive Feeding with a General-Purpose Mobile Manipulator

General-purpose mobile manipulators have the potential to serve as a versatile form of assistive technology. However, their complexity creates challenges, including the risk of being too difficult to use. We present a proof-of-concept robotic system for assistive feeding that consists of a Willow Garage PR2, a high-level web-based interface, and specialized autonomous behaviors for scooping and feeding yogurt. As a step towards use by people with disabilities, we evaluated our system with 5 able-bodied participants. All 5 successfully ate yogurt using the system and reported high rates of success for the system's autonomous behaviors. Also, Henry Evans, a person with severe quadriplegia, operated the system remotely to feed an able-bodied person. In general, people who operated the system reported that it was easy to use, including Henry. The feeding system also incorporates corrective actions designed to be triggered either autonomously or by the user. In an offline evaluation using data collected with the feeding system, a new version of our multimodal anomaly detection system outperformed prior versions.Comment: This short 4-page paper was accepted and presented as a poster on May. 16, 2016 in ICRA 2016 workshop on 'Human-Robot Interfaces for Enhanced Physical Interactions' organized by Arash Ajoudani, Barkan Ugurlu, Panagiotis Artemiadis, Jun Morimoto. It was peer reviewed by one reviewe

Web Service Testing and Usability for Mobile Learning

Based on the summary of recent renowned publications, Mobile Learning (ML) has become an emerging technology, as well as a new technique that can enhance the quality of learning. Due to the increasing importance of ML, the investigation of such impacts on the e-Science community is amongst the hot topics, which also relate to part of these research areas: Grid Infrastructure, Wireless Communication, Virtual Research Organization and Semantic Web. The above examples contribute to the demonstrations of how Mobile Learning can be applied into e-Science applications, including usability. However, there are few papers addressing testing and quality engineering issues – the core component for software engineering. Therefore, the major purpose of this paper is to present how Web Service Testing for Mobile Learning can be carried out, in addition to re-investigating the influences of the usability issue with both quantitative and qualitative research methods. Out of many mobile technologies available, the Pocket PC and Tablet PC have been chosen as the equipment; and the OMII Web Service, the 64-bit .NET e-portal and the GPS-PDA are the software tools to be used for Web Service testing