Automated Testing of Motion-based Events in Mobile Applications

Automated test case generation is one of the main challenges in testing mobile applications. This challenge becomes more complicated when the application being tested supports motion-based events. In this paper, we propose a novel, hidden Markov model (HMM)-based approach to automatically generate movement-based gestures in mobile applications. A HMM classifier is used to generate movements, which mimic a user’s behaviour in interacting with the application’s User Interface (UI). We evaluate the proposed technique on three different case studies; the evaluation indicates that the technique not only generates realistic test cases, but also achieves better code coverage when compared to randomly generated test case

Automating Software Development for Mobile Computing Platforms

Mobile devices such as smartphones and tablets have become ubiquitous in today\u27s computing landscape. These devices have ushered in entirely new populations of users, and mobile operating systems are now outpacing more traditional desktop systems in terms of market share. The applications that run on these mobile devices (often referred to as apps ) have become a primary means of computing for millions of users and, as such, have garnered immense developer interest. These apps allow for unique, personal software experiences through touch-based UIs and a complex assortment of sensors. However, designing and implementing high quality mobile apps can be a difficult process. This is primarily due to challenges unique to mobile development including change-prone APIs and platform fragmentation, just to name a few. in this dissertation we develop techniques that aid developers in overcoming these challenges by automating and improving current software design and testing practices for mobile apps. More specifically, we first introduce a technique, called Gvt, that improves the quality of graphical user interfaces (GUIs) for mobile apps by automatically detecting instances where a GUI was not implemented to its intended specifications. Gvt does this by constructing hierarchal models of mobile GUIs from metadata associated with both graphical mock-ups (i.e., created by designers using photo-editing software) and running instances of the GUI from the corresponding implementation. Second, we develop an approach that completely automates prototyping of GUIs for mobile apps. This approach, called ReDraw, is able to transform an image of a mobile app GUI into runnable code by detecting discrete GUI-components using computer vision techniques, classifying these components into proper functional categories (e.g., button, dropdown menu) using a Convolutional Neural Network (CNN), and assembling these components into realistic code. Finally, we design a novel approach for automated testing of mobile apps, called CrashScope, that explores a given android app using systematic input generation with the intrinsic goal of triggering crashes. The GUI-based input generation engine is driven by a combination of static and dynamic analyses that create a model of an app\u27s GUI and targets common, empirically derived root causes of crashes in android apps. We illustrate that the techniques presented in this dissertation represent significant advancements in mobile development processes through a series of empirical investigations, user studies, and industrial case studies that demonstrate the effectiveness of these approaches and the benefit they provide developers

GestUI: A Model-driven Method and Tool for Including Gesture-based Interaction in User Interfaces

[EN] Among the technological advances in touch-based devices, gesture-based interaction have become a prevalent feature in many application domains. Information systems are starting to explore this type of interaction. As a result, gesture specifications are now being hard-coded by developers at the source code level that hinders their reusability and portability. Similarly, defining new gestures that reflect user requirements is a complex process. This paper describes a model-driven approach to include gesture-based interaction in desktop information systems. It incorporates a tool prototype that captures user-sketched multi-stroke gestures and transforms them into a model by automatically generating the gesture catalogue for gesture-based interaction technologies and gesture-based user interface source codes. We demonstrated our approach in several applications ranging from case tools to form-based information systems.This work was supported by SENESCYT and Universidad de Cuenca from Ecuador, and received financial support from Generalitat Valenciana under Project IDEO (PROMETEOII/2014/039).Parra-González, LO.; España Cubillo, S.; Pastor López, O. (2016). GestUI: A Model-driven Method and Tool for Including Gesture-based Interaction in User Interfaces. Complex Systems Informatics and Modeling Quarterly. 6:73-92. https://doi.org/10.7250/csimq.2016-6.05S7392

Towards More Accessible Android Applications : An Actionable Accessibility Checklist for Android Developers

Approximately 15% of the world's population has a disability or impairment. They need mobile applications to be developed with accessibility in mind to use them effectively. However, most mobile applications are not accessible because developers often lack the knowledge or skills to create them. Additionally, there are differences between operating systems, and Android apps have more accessibility issues compared to iOS. The goal of this master’s thesis is to understand how Android developers could develop more accessible apps than currently available. To reach this goal, the method of the study is design science research method. The outcome of this thesis is an artifact, a checklist for Android developers to use when developing an app to test app’s accessibility. For the development of the checklist prior research was used, and with its help, the first version of the checklist was created. The initial version was evaluated with interviews with five Android developers. Interview answers were analyzed with thematization. The results of the analysis contributed to a prototype, which was a website with the checklist and accompanying learning material and instructions. The usefulness and functionality of the prototype was tested with a questionnaire. The questions in the questionnaire were related to the usefulness of the checklist and how to improve the checklist. Thematization was used for the answers of the questionnaire as well. The analyzed answers were used for improving the prototype and constructing the final checklist. It consists of six themes, which all have from one to six checks. The themes were related to automated tools, visual elements like images, alternative ways of navigation, screen orientation, structure of the user interface, and screen magnification. In addition to these themes, the list has checks that might be out of the developer’s control, but worth checking. These checks are categorized into four themes: Visual elements like images, use of color, screen orientation, and audio- and video content. In addition to checks, the website with the checklist has instructions for conducting the checks, and additional material for learning about accessibility.Arvioilta 15 % maailman väestöstä on jonkinlainen vamma tai toimintarajoite. Jotta he pystyisivät käyttämään mobiilisovelluksia, täytyy sovellukset rakentaa saavutettavasti. Tällä hetkellä iso osa mobiilisovelluksista ei ole saavutettavia, ja yksi syy siihen on sovellusten kehittäjien osaamisen ja tiedon puute. Käyttöjärjestelmien välillä on eroja, ja kahdesta suosituimmasta mobiilikäyttöjärjestelmästä Android-sovelluksissa on selkeästi enemmän saavutettavuuspuutteita. Tämän tutkielman tavoitteena on ymmärtää, miten Android-kehittäjät voivat kehittää sovelluksia, jotka ovat saavutettavampia kuin nykyiset sovellukset. Tavoitteen saavuttamiseksi tutkimuksessa käytettiin suunnittelutieteellistä tutkimusmenetelmää, ja sen prosessimallia. Työn tuloksena syntyi tarkistuslista Android-sovelluksen testauslista, jonka avulla sovelluskehittäjä voi testata sovelluksensa saavutettavuuden eri aspekteja. Tarkistuslistan kehityksessä käytettiin hyväksi aikaisempaa tutkimusta, ja sen avulla luotiin listan ensimmäinen versio. Tämän jälkeen listan käytettävyyttä ja hyödyllisyyttä arvioitiin viiden Android-kehittäjän haastatteluilla. Haastattelut analysoitiin teemoittelun avulla. Analyysin pohjalta kehitettiin prototyyppi, sivusto, jolla tarkistuslista sekä sitä tukeva ohjeistusmateriaali on saatavilla. Tämän prototyypin hyödyllisyyttä ja toimivuutta testattiin kyselylomakkeen avulla, jossa Android-kehittäjiltä kysyttiin listan hyödyllisyydestä, ja siitä, miten sitä voisi parantaa. Tämäkin aineisto analysoitiin teemoittelun avulla. Näiden kehitysehdotusten perusteella luotiin lopullinen tarkistuslista, joka sisältää kuusi teemaa, joiden alla on yhdestä kuuteen tarkistuskohtaa. Nämä teemat liittyvät automatisoituihin työkaluihin, visuaalisiin elementteihin (kuten kuvat), erilaisiin tapoihin navigoida, näytön suuntaan, käyttöliittymän rakenteeseen sekä näytön kokoon ja sen suurentamiseen. Lisäksi listalla on neljä teemaa, joihin kehittäjä ei välttämättä pysty vaikuttamaan, mutta jotka kannattaa testata – etenkin jos niihin pystyy vaikuttamaan. Nämä teemat liittyvät osaltaan visuaalisiin elementteihin (kuten kuvat), värien käyttöön, näytön suuntaan sekä audio- ja videosisältöön. Tarkistuslistan lisäksi sivusto, jolla tarkistuslista esitellään, sisältää ohjeistuksen tarkistusten tekemiseen sekä lisämateriaalia saavutettavuudesta oppimiseen

Cloud Platform for Research Crowdsourcing in Mobile Testing

Mobile application testing and testing over a cloud are two highly topical fields nowadays. Mobile testing presents specific test activities, including verification of an application against a variety of heterogeneous smartphone models and versions of operating systems (OS), build distribution and test team management, monitoring and user experience analytics of an application in production, etc. Cloud benefits are widely used to support all these activities. This study conducts in-depth analyses of existing cloud services for mobile testing and addresses their weaknesses regarding research purposes and testing needs of the critical and business-critical mobile applications.   During this study, a Cloud Testing of Mobile Systems (CTOMS) framework for effective research crowdsourcing in mobile testing was developed. The framework is presented as a lightweight and easily scalable distributed system that provides a cloud service to run tests on a variety of remote mobile devices. CTOMS provides implementation of two novel functionalities that are demanded by advanced investigations in mobile testing. First, it allows full multidirectional testing, which provides the opportunities to test an application on different devices and/or OS versions, and new device models or OS versions for their compatibility with the most popular applications in the market, or just legacy critical apps, etc. Second, CTOMS demonstrates the effective integration of the appropriate testing techniques for mobile development within such a service. In particular, it provides a user with suggestions about coverage of configurations to test on using combinatorial approaches like a base choice, pair-wise, and t-way. The current CTOMS version supports automated functional testing of Android applications and detection of defects in the user interface (UI). This has a great value because requirements for UI and user experience are high for any modern mobile application.    The fundamental analysis of possible test types and techniques using a system like CTOMS was conducted, and ways of possible enhancements and extensions of functionality for possible research are listed. The first case studies prove the work of implemented novel concepts, their usefulness, and their convenience for experiments in mobile testing. The overall work proves that a study of cloud mobile testing is feasible even with small research resources.  M.S

Proceedings of the 2012 Workshop on Ambient Intelligence Infrastructures (WAmIi)

This is a technical report including the papers presented at the Workshop on Ambient Intelligence Infrastructures (WAmIi) that took place in conjunction with the International Joint Conference on Ambient Intelligence (AmI) in Pisa, Italy on November 13, 2012. The motivation for organizing the workshop was the wish to learn from past experience on Ambient Intelligence systems, and in particular, on the lessons learned on the system architecture of such systems. A significant number of European projects and other research have been performed, often with the goal of developing AmI technology to showcase AmI scenarios. We believe that for AmI to become further successfully accepted the system architecture is essential

Proceedings of the 2012 Workshop on Ambient Intelligence Infrastructures (WAmIi)

This is a technical report including the papers presented at the Workshop on Ambient Intelligence Infrastructures (WAmIi) that took place in conjunction with the International Joint Conference on Ambient Intelligence (AmI) in Pisa, Italy on November 13, 2012. The motivation for organizing the workshop was the wish to learn from past experience on Ambient Intelligence systems, and in particular, on the lessons learned on the system architecture of such systems. A significant number of European projects and other research have been performed, often with the goal of developing AmI technology to showcase AmI scenarios. We believe that for AmI to become further successfully accepted the system architecture is essential

Using natural user interfaces to support synchronous distributed collaborative work

Synchronous Distributed Collaborative Work (SDCW) occurs when group members work together at the same time from different places together to achieve a common goal. Effective SDCW requires good communication, continuous coordination and shared information among group members. SDCW is possible because of groupware, a class of computer software systems that supports group work. Shared-workspace groupware systems are systems that provide a common workspace that aims to replicate aspects of a physical workspace that is shared among group members in a co-located environment. Shared-workspace groupware systems have failed to provide the same degree of coordination and awareness among distributed group members that exists in co-located groups owing to unintuitive interaction techniques that these systems have incorporated. Natural User Interfaces (NUIs) focus on reusing natural human abilities such as touch, speech, gestures and proximity awareness to allow intuitive human-computer interaction. These interaction techniques could provide solutions to the existing issues of groupware systems by breaking down the barrier between people and technology created by the interaction techniques currently utilised. The aim of this research was to investigate how NUI interaction techniques could be used to effectively support SDCW. An architecture for such a shared-workspace groupware system was proposed and a prototype, called GroupAware, was designed and developed based on this architecture. GroupAware allows multiple users from distributed locations to simultaneously view and annotate text documents, and create graphic designs in a shared workspace. Documents are represented as visual objects that can be manipulated through touch gestures. Group coordination and awareness is maintained through document updates via immediate workspace synchronization, user action tracking via user labels and user availability identification via basic proxemic interaction. Members can effectively communicate via audio and video conferencing. A user study was conducted to evaluate GroupAware and determine whether NUI interaction techniques effectively supported SDCW. Ten groups of three members each participated in the study. High levels of performance, user satisfaction and collaboration demonstrated that GroupAware was an effective groupware system that was easy to learn and use, and effectively supported group work in terms of communication, coordination and information sharing. Participants gave highly positive comments about the system that further supported the results. The successful implementation of GroupAware and the positive results obtained from the user evaluation provides evidence that NUI interaction techniques can effectively support SDCW

On Supporting Android Software Developers And Testers

Users entrust mobile applications (apps) to help them with different tasks in their daily lives. However, for each app that helps to finish a given task, there are a plethora of other apps in popular marketplaces that offer similar or nearly identical functionality. This makes for a competitive market where users will tend to favor the highest quality apps in most cases. Given that users can easily get frustrated by apps which repeatedly exhibit bugs, failures, and crashes, it is imperative that developers promptly fix problems both before and after the release. However, implementing and maintaining high quality apps is difficult due to unique problems and constraints associated with the mobile development process such as fragmentation, quick feature changes, and agile software development. This dissertation presents an empirical study, as well as several approaches for developers, testers and designers to overcome some of these challenges during the software development life cycle. More specifically, first we perform an in-depth analysis of developers’ needs on automated testing techniques. This included surveying 102 contributors of open source Android projects about practices for testing their apps. The major findings from this survey illustrate that developers: (i) rely on usage models for designing test app cases, (ii) prefer expressive automated generated test cases organized around use cases, (iii) prefer manual testing over automation due to reproducibility issues, and (iv) do not perceive that code coverage is an important measure of test case quality. Based on the findings from the survey, this dissertation presents several approaches to support developers and testers of Android apps in their daily tasks. In particular, we present the first taxonomy of faults in Android apps. This taxonomy is derived from a manual analysis of 2,023 software artifacts extracted from six different sources (e.g., stackoverflow and bug reports). The taxonomy is divided into 14 categories containing 262 specific types. Then, we derived 38 Android-specific mutation operators from the taxonomy. Additionally, we implemented the infrastructure called MDroid+ that automatically introduces mutations in Android apps. Third, we present a practical automation for crowdsourced videos of mobile apps called V2S. This solution automatically translates video recordings of mobile executions into replayable user scenarios. V2S uses computer vision and adopts deep learning techniques to identify user interactions from video recordings that illustrate bugs or faulty behaviors in mobile apps. Last but not least, we present an approach that aims at supporting the maintenance process by facilitating the way users report bugs for Android apps. It comprises the interaction between an Android and a web app that assist the reporter by automatically collecting relevant information