Flight control system design factors for applying automated testing techniques

Automated validation of flight-critical embedded systems is being done at ARC Dryden Flight Research Facility. The automated testing techniques are being used to perform closed-loop validation of man-rated flight control systems. The principal design features and operational experiences of the X-29 forward-swept-wing aircraft and F-18 High Alpha Research Vehicle (HARV) automated test systems are discussed. Operationally applying automated testing techniques has accentuated flight control system features that either help or hinder the application of these techniques. The paper also discusses flight control system features which foster the use of automated testing techniques

Towards Efficient Record and Replay: A Case Study in WeChat

WeChat, a widely-used messenger app boasting over 1 billion monthly active users, requires effective app quality assurance for its complex features. Record-and-replay tools are crucial in achieving this goal. Despite the extensive development of these tools, the impact of waiting time between replay events has been largely overlooked. On one hand, a long waiting time for executing replay events on fully-rendered GUIs slows down the process. On the other hand, a short waiting time can lead to events executing on partially-rendered GUIs, negatively affecting replay effectiveness. An optimal waiting time should strike a balance between effectiveness and efficiency. We introduce WeReplay, a lightweight image-based approach that dynamically adjusts inter-event time based on the GUI rendering state. Given the real-time streaming on the GUI, WeReplay employs a deep learning model to infer the rendering state and synchronize with the replaying tool, scheduling the next event when the GUI is fully rendered. Our evaluation shows that our model achieves 92.1% precision and 93.3% recall in discerning GUI rendering states in the WeChat app. Through assessing the performance in replaying 23 common WeChat usage scenarios, WeReplay successfully replays all scenarios on the same and different devices more efficiently than the state-of-the-practice baselines

The Integration of Machine Learning into Automated Test Generation: A Systematic Mapping Study

Context: Machine learning (ML) may enable effective automated test generation. Objective: We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges. Methods: We perform a systematic mapping on a sample of 102 publications. Results: ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property-based, and expected output oracles. Supervised learning - often based on neural networks - and reinforcement learning - often based on Q-learning - are common, and some publications also employ unsupervised or semi-supervised learning. (Semi-/Un-)Supervised approaches are evaluated using both traditional testing metrics and ML-related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. Conclusion: Work-to-date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed - and how they are applied - benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.Comment: Under submission to Software Testing, Verification, and Reliability journal. (arXiv admin note: text overlap with arXiv:2107.00906 - This is an earlier study that this study extends

Engine Fuel Injection Timing : A Design for an Automatic Verification System

This thesis describes the development of an automatic testing system for the timing of the fuel injection of a 4-stroke engine. The fuel injection timing is managed by an electronic engine control unit which has a distributed modular design. New software and hardware updates are released every few months for the engine control unit. Furthermore, fuel injection timing must be tested for each new software release, because incorrect timing could potentially lead to engine failure. Thus, automating this frequent testing procedure, which can take 2–5 days manually, is expected to save both time and money. Therefore, the object of this work is to develop a design of an automatic fuel injection timing testing system. There are already abundant scientific studies available related to fuel injection timing and engine control unit. The majority of these studies in the literature review cover various topics about the effects of alternative injection technologies and fuels. A limited number of them comprise the subject of automatic fuel injection timing. Design science was chosen as the research method because of its suitability for product development projects. The most important research question is what the design architecture must be like for testing injection timing. This work started with a comprehensive analysis of the different factors that could affect the design. Underlying motivation for developing an automatic testing system, stakeholders involved, alternative ways for testing implementation, and various other points of view were covered. After defining the system requirements, the setup was built to measure the timing of fuel injection pulses from the engine control unit, which utilized the National Instruments Compact RIO hardware and software programmed with LabVIEW. This program automatically generates an Excel report of the timing test. The design of a testing system architecture that would allow measurements to be made from any of the 112 fuel injection terminals of the control unit was successfully developed. Measurements performed with Compact RIO hardware proved to be accurate and could determine the crankshaft angle with the required accuracy. The accuracy of the testing system was ±5 μs. Next, the development of communication between the testing hardware and the engine control unit’s configuration software was identified as the most important issue for future development of the testing system. The proposed testing system principle is probably feasible for developing any further automatic testing systems for any electric engine control unit in which fuel injection timing needs to be verified. Moreover, Compact RIO hardware and LabVIEW software can be recommended as a tool for developing similar verification systems because they are relatively easy to use, flexible, reliable, and capable of high-speed measurements.Tämä diplomityö kuvaa automaattisen testausjärjestelmän kehittämistä nelitahtimoottorin polttoaineen ruiskutussignaalien ajoitukselle. Ruiskutuksen ajoitusta hallitaan sähköisellä moottorinohjausyksiköllä, millä on hajautettu modulaarinen rakenne. Uusia moottorinohjausyksikön ohjelmisto- ja laitteistoversioita julkaistaan muutaman kuukauden välein. Polttoaineensyötön oikea ajoitus täytyy testata aina, kun uusia versioita julkaistaan, koska väärä ajoitus saattaa aiheuttaa moottorihäiriön. Usein toistuvan testauksen automatisoinnin odotetaan lyhentävän siihen käytettävää aikaa ja kustannuksia merkittävästi, mikä manuaalisesti tehtynä voi kestää 2–5 päivää. Työn tavoitteena on kehittää suunnitelma automaattisesta testausjärjestelmästä polttoaineen ruiskutuksen ajoitukselle. Polttoaineenruiskutukseen ja moottorinohjausyksiköihin liittyviä tieteellisiä julkaisuja on saatavilla runsaasti. Suurin osa kirjallisuuskatsauksessa käsitellyistä tutkimuksista kattaa eri aiheita vaihtoehtoisten ruiskutustekniikoiden ja polttoaineiden vaikutuksista polttomoottoriin. Vain muutama niistä käsittelee polttoaineensyötön automaattista testausta. Tutkimusmenetelmäksi valittiin suunnittelutiede, koska se soveltuu hyvin tuotekehitysprojekteihin. Tärkein tutkimuskysymys on: ”Minkälainen järjestelmän arkkitehtuurin täytyy olla ruiskutuksen ajoituksen testaamista varten?” Kysymyksen tutkiminen aloitettiin analysoimalla perusteellisesti eri tekijöitä, jotka voisivat vaikuttaa toteutukseen. Mikä on se perimmäinen syy miksi automaattinen testausjärjestelmä halutaan kehittää, mukana olevat sidosryhmät, vaihtoehtoiset toteutustavat sekä useita muita näkökulmia huomioitiin. Järjestelmävaatimusten määrittelyn jälkeen rakennettiin koelaite, jolla mitattiin polttoaineensyötön pulssien ajoitusta moottorinohjausyksiköstä, mikä hyödynsi National Instruments Compact RIO laitteistoa ja ohjelmistoa mikä kehitettiin LabVIEW -kehitysympäristössä. Ohjelma luo automaattisesti Excel raportin ajoitustesteistä. Onnistuneesti luotiin testausjärjestelmän arkkitehtuuri, mikä mahdollistaa mittausten tekemisen mistä tahansa hajautetun moottorinohjausyksikön 112 polttoaineensyötön liittimestä. Compact RIO laitteistolla tehdyt mittaukset osoittautuivat tarkoiksi ja se pystyy määrittämään kampiakselin kulman vaaditulla tarkkuudella. Testausjärjestelmän tarkkuus oli ±5 μs. Kommunikaation kehittäminen testauslaitteiston ja moottorinohjausyksikön konfigurointi ohjelmiston välille tunnistettiin kaikkein tärkeimmäksi asiaksi testausjärjestelmän jatkokehitykselle. Ehdotettu arkkitehtuuri on todennäköisesti sopiva ratkaisu automaattisen testausjärjestelmän kehittämiseksi mille tahansa sähköiselle moottorinohjausyksikölle, jonka polttoaineen ruiskutuksen ajoitus halutaan varmentaa. Lisäksi Compact RIO laitteistoa ja LabVIEW ohjelmistoa voidaan suositella työkaluiksi vastaavien testausjärjestelmien kehittämiseen koska ne ovat kohtuullisen helppokäyttöisiä, joustavia, luotettavia ja pystyvät nopeisiin mittauksiin

Designing for designers: Towards the development of accessible ICT products and services using the VERITAS framework

Among key design practices which contribute to the development of inclusive ICT products and services is user testing with people with disabilities. Traditionally, this involves partial or minimal user testing through the usage of standard heuristics, employing external assisting devices, and the direct feedback of impaired users. However, efficiency could be improved if designers could readily analyse the needs of their target audience. The VERITAS framework simulates and systematically analyses how users with various impairments interact with the use of ICT products and services. Findings show that the VERITAS framework is useful to designers, offering an intuitive approach to inclusive design.The work presented in this article forms part of VERITAS, which is funded by the European Commission's 7th Framework Programme (FP7) (grant agreement # 247765 FP7-ICT-2009.7.2)

Keeping Context In Mind: Automating Mobile App Access Control with User Interface Inspection

Recent studies observe that app foreground is the most striking component that influences the access control decisions in mobile platform, as users tend to deny permission requests lacking visible evidence. However, none of the existing permission models provides a systematic approach that can automatically answer the question: Is the resource access indicated by app foreground? In this work, we present the design, implementation, and evaluation of COSMOS, a context-aware mediation system that bridges the semantic gap between foreground interaction and background access, in order to protect system integrity and user privacy. Specifically, COSMOS learns from a large set of apps with similar functionalities and user interfaces to construct generic models that detect the outliers at runtime. It can be further customized to satisfy specific user privacy preference by continuously evolving with user decisions. Experiments show that COSMOS achieves both high precision and high recall in detecting malicious requests. We also demonstrate the effectiveness of COSMOS in capturing specific user preferences using the decisions collected from 24 users and illustrate that COSMOS can be easily deployed on smartphones as a real-time guard with a very low performance overhead.Comment: Accepted for publication in IEEE INFOCOM'201

Improving Automated Software Testing while re-engineering legacy systems in the absence of documentation

Legacy software systems are essential assets that contain an organizations' valuable business logic. Because of outdated technologies and methods used in these systems, they are challenging to maintain and expand. Therefore, organizations need to decide whether to redevelop or re-engineer the legacy system. Although in most cases, re-engineering is the safer and less expensive choice, it has risks such as failure to meet the expected quality and delays due to testing blockades. These risks are even more severe when the legacy system does not have adequate documentation. A comprehensive testing strategy, which includes automated tests and reliable test cases, can substantially reduce the risks. To mitigate the hazards associated with re-engineering, we have conducted three studies in this thesis to improve the testing process. Our rst study introduces a new testing model for the re-engineering process and investigates test automation solutions to detect defects in the early re-engineering stages. We implemented this model on the Cold Region Hydrological Model (CRHM) application and discovered bugs that would not likely have been found manually. Although this approach helped us discover great numbers of software defects, designing test cases is very time-consuming due to the lack of documentation, especially for large systems. Therefore, in our second study, we investigated an approach to generate test cases from user footprints automatically. To do this, we extended an existing tool to collect user actions and legacy system reactions, including database and le system changes. Then we analyzed the data based on the order of user actions and time of them and generated human-readable test cases. Our evaluation shows that this approach can detect more bugs than other existing tools. Moreover, the test cases generated using this approach contain detailed oracles that make them suitable for both black-box and white-box testing. Many scienti c legacy systems such as CRHM are data-driven; they take large amounts of data as input and produce massive data after applying mathematical models. Applying test cases and nding bugs is more demanding when we are dealing with large amounts of data. Hence in our third study, we created a comparative visualization tool (ComVis) to compare a legacy system's output after each change. Visualization helps testers to nd data issues resulting from newly introduced bugs. Twenty participants took part in a user study in which they were asked to nd data issued using ComVis and embedded CRHM visualization tool. Our user study shows that ComVis can nd 51% more data issues than embedded visualization tools in the legacy system can. Also, results from the NASA-TLX assessment and thematic analysis of open-ended questions about each task show users prefer to use ComVis over the built-in visualization tool. We believe our introduced approaches and developed systems will signi cantly reduce the risks associated with the re-engineering process. i

Voice recognition system for Massey University Smarthouse : a thesis presented in partial fulfilment of the requirements for the degree of Master of Engineering in Information Engineering at Massey University

The concept of a smarthouse aims to integrate technology into houses to a level where most daily tasks are automated and to provide comfort, safety and entertainment to the house residents. The concept is mainly aimed at the elderly population to improve their quality of life. In order to maintain a natural medium of communication, the house employs a speech recognition system capable of analysing spoken language, and extracting commands from it. This project focuses on the development and evaluation of a windows application developed with a high level programming language which incorporates speech recognition technology by utilising a commercial speech recognition engine. The speech recognition system acts as a hub within the Smarthouse to receive and delegate user commands to different switching and control systems. Initial trails were built using Dragon Naturally Speaking as the recognition engine. However that proved inappropriate for use in the Smarthouse project as it is speaker dependent and requires each user to train it with his/her own voice. The application now utilizes the Microsoft Speech Application Programming Interface (SAPI), a software layer which sits between applications and speech engines and the Microsoft Speech Recognition Engine, which is freely distributed with some Microsoft products. Although Dragon Naturally Speaking offers better recognition for dictation, MS engine can be optimized using Context Free Grammar (CFG) to give enhanced recognition in the intended application. The application is designed to be speaker independent and can handle continuous speech. It connects to a database oriented expert system to carry out full conversations with the users. Audible prompts and confirmations are achieved through speech synthesis using any SAPI compliant text to speech engine. Other developments focused on designing a telephony system using Microsoft Telephony Application Programming Interface (TAPI). This allows the house to be remotely controlled from anywhere in the world. House residents will be able to call their house from any part of the world and regardless of their location, the house will be able to respond to and fulfil their commands

Automated GUI performance testing

A significant body of prior work has devised approaches for automating the functional testing of interactive applications. However, little work exists for automatically testing their performance. Performance testing imposes additional requirements upon GUI test automation tools: the tools have to be able to replay complex interactive sessions, and they have to avoid perturbing the application's performance. We study the feasibility of using five Java GUI capture and replay tools for GUI performance test automation. Besides confirming the severity of the previously known GUI element identification problem, we also describe a related problem, the temporal synchronization problem, which is of increasing importance for GUI applications that use timer-driven activity. We find that most of the tools we study have severe limitations when used for recording and replaying realistic sessions of real-world Java applications and that all of them suffer from the temporal synchronization problem. However, we find that the most reliable tool, Pounder, causes only limited perturbation and thus can be used to automate performance testing. Based on an investigation of Pounder's approach, we further improve its robustness and reduce its perturbation. Finally, we demonstrate in a set of case studies that the conclusions about perceptible performance drawn from manual tests still hold when using automated tests driven by Pounder. Besides the significance of our findings to GUI performance testing, the results are also relevant to capture and replay-based functional GUI test automation approache