An Investigation into Improving Test Effectiveness for Embedded Software

This thesis reports on the investigation of the effectiveness of software testing on embedded systems. The aim was to improve confidence in the current methods employed or to find new methods which could improve the hit rate of defects found before software is sent to a customer. We investigate previous work into software testing effectives and various black box testing methods. There are various Black Box testing methodologies that can be employed to detect errors in systems with varying degrees of success. In this thesis we investigate the transformation of the white box testing technique of Definition Use (DU) Path testing using a RESOLVE like specification, to be applied as black box test method. We do not use RESOLVE it self, instead we defined our own method of automatic test generation based on the principles of RESOLVE. Then we compare this method to more commonly used requirements driven test selection, and pure boundary value analysis (BVA) testing techniques. The results reported in this thesis indicate that BVA and DU test selection methods create tests that are covered by unit and integration tests. The current requirements driven test cases create tests with a combination of features working in tandem. It was found that combination of features was more likely to find defects because developers tests had a lesser focus on this area. The tests generated by the BVA and DU test selection methods did not find any defects that their respective methods were intended to find. This is due to the development team already having tests that covered these areas and defects had been fixed before system tests could be run. Based on the fact that the current test selection methods find defects and the methods we investigated do not, this adds confidence that the system test approach to testing is effective. The investigation of defects found showed that timing related errors are common and that a test selection method designed to find timing related defects would be worth investigating. The investigation also revealed a useful method in automatic generation of test cases. The RESOLVE like specification was used to apply a DU testing as a black box test method. This method showed to be more time efficient at generating test cases than the existing requirements driven approach. Although the test cases did not reveal significant defects, due to the overlap with integration testing, it could be a useful method for developers to generate test cases

Mbed OS regression test selection and optimization

Abstract. Testing is a fundamental building block in the identification of bugs, errors and defects in both hardware and software. Effective testing of large projects requires automated testing, test selection and test optimization. Using CI (Continuous Integration) tools, and test selection and optimization techniques reduce development time and increase productivity. The prioritization, selection and minimization of tests are well-known problems in software testing. Arm Mbed OS is a free, open-source embedded operating system designed specifically for the “things” in the IoT (Internet of Things). This thesis researches regression test selection (RTS) and optimization techniques (RTO). The main focus of the thesis is to develop a set of effective automated safe RTS (mbedRTS) and RTO (mbedRTO) techniques for Mbed OS pull request (PR) testing. This thesis refers to the set of developed techniques as Mbed OS regression test techniques (MbedRTT), also known as Mbed OS Smart Tester. The empirical analysis of the researched, and developed MbedRTT techniques show promising results. Several developed MbedRTT techniques have already been adopted in Mbed OS Jenkins CI.Mbed OS -regressiotestien valinta ja optimointi. Tiivistelmä. Testaus on olennainen tekijä vikojen ja virheiden tunnistamisessa sekä ohjelmistossa että laitteistossa. Isojen projektien tehokas testaaminen vaatii automaattista testausta, testien valintaa ja testien optimointia. Jatkuvan integraation (engl. continuous integration) työkalut, testien valintatekniikat ja testien optimointitekniikat lyhentävät kehitykseen kuluvaa aikaa ja kasvattavat tuottavuutta. Testien priorisointi, valinta ja minimointi ovat tunnettuja ongelmia ohjelmistotestauksessa. Arm Mbed OS on ilmainen avoimen lähdekoodin sulautettu käyttöjärjestelmä, joka on tarkoitettu erityisesti “asioille” asioiden Internetissä (engl. Internet of Things). Tässä työssä tutkitaan regressiotestauksen valinta- ja optimointimenetelmiä. Tämän työn päätehtävä on kehittää tehokkaita ja turvallisia valinta- (mbedRTS) ja optimointimenetelmiä (mbedRTO) Mbed OS pull request:ien regressiotestaukseen. Mbed OS -regressiotestausmenetelmillä (MbedRTT) viitataan tässä työssä kehitettyihin regressiotestausmenetelmiin, jotka tunnetaan myös nimellä Mbed OS älykäs testaaja (engl. Mbed OS Smart Tester). Tutkittujen ja kehitettyjen MbedRTT-tekniikoiden empiirisen analyysin tulos näyttää lupaavalta. Mbed OS Jenkins CI:ssä on jo otettu käyttöön useita kehitettyjä MbedRTT-tekniikoita

Perbandingan Kinerja Antara Gatling dan Apache JMeter pada Uji Beban RESTful API

This research explores and compares the performance of two popular load testing tools, namely Gatling and Apache JMeter, with a focus on API performance testing. The rapid growth in web and mobile application development highlights the urgent need to ensure optimal API performance. This research was conducted to provide in-depth insight into the advantages and disadvantages of these two testing tools through the use of similar testing scenarios. The experimental method involves implementing test scenarios that include load variations and high demands on both devices. The main parameters observed include API response time, throughput, and latency. In-depth analysis was carried out on the data obtained to evaluate the reliability and efficiency of each tool. The results of this research provide a comprehensive understanding of the performance of Gatling and Apache JMeter in the context of API performance testing. These findings can provide practical guidance for software developers and testing practitioners in selecting load testing tools that suit their project needs. Recommendations for future research include expanding exploration of other load testing tools, comparison with more complex test scenarios, and integration with performance monitoring tools for more holistic analysis. Thus, this research is expected to make a significant contribution to the understanding and selection of effective load testing tools in web and mobile application development. &nbsp

Constraint-Guided Test Execution Scheduling: An Experience Report at ABB Robotics

Automated test execution scheduling is crucial in modern software development environments, where components are frequently updated with changes that impact their integration with hardware systems. Building test schedules, which focus on the right tests and make optimal use of the available resources, both time and hardware, under consideration of vast requirements on the selection of test cases and their assignment to certain test execution machines, is a complex optimization task. Manual solutions are time-consuming and often error-prone. Furthermore, when software and hardware components and test scripts are frequently added, removed or updated, static test execution scheduling is no longer feasible and the motivation for automation taking care of dynamic changes grows. Since 2012, our work has focused on transferring technology based on constraint programming for automating the testing of industrial robotic systems at ABB Robotics. After having successfully transferred constraint satisfaction models dedicated to test case generation, we present the results of a project called DynTest whose goal is to automate the scheduling of test execution from a large test repository, on distinct industrial robots. This paper reports on our experience and lessons learned for successfully transferring constraint-based optimization models for test execution scheduling at ABB Robotics. Our experience underlines the benefits of a close collaboration between industry and academia for both parties.Comment: SafeComp 202

Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration

Testing in Continuous Integration (CI) involves test case prioritization, selection, and execution at each cycle. Selecting the most promising test cases to detect bugs is hard if there are uncertainties on the impact of committed code changes or, if traceability links between code and tests are not available. This paper introduces Retecs, a new method for automatically learning test case selection and prioritization in CI with the goal to minimize the round-trip time between code commits and developer feedback on failed test cases. The Retecs method uses reinforcement learning to select and prioritize test cases according to their duration, previous last execution and failure history. In a constantly changing environment, where new test cases are created and obsolete test cases are deleted, the Retecs method learns to prioritize error-prone test cases higher under guidance of a reward function and by observing previous CI cycles. By applying Retecs on data extracted from three industrial case studies, we show for the first time that reinforcement learning enables fruitful automatic adaptive test case selection and prioritization in CI and regression testing.Comment: Spieker, H., Gotlieb, A., Marijan, D., & Mossige, M. (2017). Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration. In Proceedings of 26th International Symposium on Software Testing and Analysis (ISSTA'17) (pp. 12--22). AC

Visualizing test diversity to support test optimisation

Diversity has been used as an effective criteria to optimise test suites for cost-effective testing. Particularly, diversity-based (alternatively referred to as similarity-based) techniques have the benefit of being generic and applicable across different Systems Under Test (SUT), and have been used to automatically select or prioritise large sets of test cases. However, it is a challenge to feedback diversity information to developers and testers since results are typically many-dimensional. Furthermore, the generality of diversity-based approaches makes it harder to choose when and where to apply them. In this paper we address these challenges by investigating: i) what are the trade-off in using different sources of diversity (e.g., diversity of test requirements or test scripts) to optimise large test suites, and ii) how visualisation of test diversity data can assist testers for test optimisation and improvement. We perform a case study on three industrial projects and present quantitative results on the fault detection capabilities and redundancy levels of different sets of test cases. Our key result is that test similarity maps, based on pair-wise diversity calculations, helped industrial practitioners identify issues with their test repositories and decide on actions to improve. We conclude that the visualisation of diversity information can assist testers in their maintenance and optimisation activities

Information selection and belief updating in hypothesis evaluation

This thesis is concerned with the factors underlying both selection and use of evidence in the testing of hypotheses. The work it describes examines the role played in hypothesis evaluation by background knowledge about the probability of events in the environment as well as the influence of more general constraints. Experiments on information choice showed that subjects were sensitive both to explicitly presented probabilistic information and to the likelihood of evidence with regard to background beliefs. It is argued - in contrast with other views in the literature - that subjects' choice of evidence to test hypotheses is rational allowing for certain constraints on subjects' cognitive representations. The majority of experiments in this thesis, however, are focused on the issue of how the information which subjects receive when testing hypotheses affects their beliefs. A major finding is that receipt of early information creates expectations which influence the response to later information. This typically produces a recency effect in which presenting strong evidence after weak evidence affects beliefs more than if the same evidence is presented in the opposite order. These findings run contrary to the view of the belief revision process which is prevalent in the literature in which it is generally assumed that the effects of successive pieces of information are independent. The experiments reported here also provide evidence that processes of selective attention influence evidence interpretation: subjects tend to focus on the most informative part of the evidence and may switch focus from one part of the evidence to another as the task progresses. in some cases, such changes of attention can eliminate the recency effect. In summary, the present research provides new evidence about the role of background beliefs, expectations and cognitive constraints in the selection and use of information to test hypotheses. Several new findings emerge which require revision to current accounts of information integration in the belief revision literature.Faculty of Human Sciences at the University of Plymout

Overcoming barriers to experimentation in business-to-business living labs

Business-to-business (B2B) living lab projects have been mentioned in different areas of academic research, but the innovation management literature requires deeper analysis of their potential opportunities and challenges. Real-life experimentation is a key requirement for living labs as it enables deeper insights in the potential success of innovations. However, the literature has not provided insights on how living lab projects can implement real-life experimentation in B2B innovation projects and does not describe appropriate conditions for experimentation in these settings. In this study, we identified three main barriers preventing real-life experimentation in B2B living lab projects: the technological complexity, the need for integration, and the difficulty in identifying testers. The barriers are discussed in detailed and potential solutions are provided to help overcome these barriers and stimulate the adoption of real-life experimentation in B2B innovation projects