Massively Parallel, Highly Efficient, but What about the Test Suite Quality? Applying Mutation Testing to GPU Programs

Thanks to rapid advances in programmability and performance, GPUs have been widely applied in High Performance Computing (HPC) and safety-critical domains. As such, quality assurance of GPU applications has gained increasing attention. This brings us to mutation testing, a fault-based testing technique that assesses the test suite quality by systematically introducing small artificial faults. It has been shown to perform well in exposing faults. In this paper, we investigate whether GPU programming can benefit from mutation testing. In addition to conventional mutation operators, we propose nine GPU-specific mutation operators based on the core syntax differences between CPU and GPU programming. We conduct a preliminary study on six CUDA systems. The results show that mutation testing can effectively evaluate the test quality of GPU programs: conventional mutation operators can guide the engineers to write simple direct tests, while GPU-specific mutation operators can lead to more intricate test cases which are better at revealing GPU-specific weaknesses. </p

How Does This New Developer Test Fit In? A Visualization to Understand Amplified Test Cases

Developer testing, the practice of software engineers programmatically checking that their own components behave as they expect, has become the norm in today's software projects. With the constantly growing size and complexity of software projects and with the rise of automated test generation tools, understanding a test case is becoming more and more important compared to writing test cases from scratch.This holds especially in the area of developer-centric test amplification, where a tool automatically generates new test cases to improve a developer-maintained test suite. To investigate how visualization can help developers understand and judge test cases, we present the TESTIMPACTGRAPH, a visualization of the call tree and coverage impact of a JUnit test case proposed for amplification. It empowers the developer to drill down into the behavior of a test case, as well as providing them a clear view on how the proposed test case contributes to the coverage of the overall test suite. In a think-aloud study we investigate which information developers seek from the TESTIMPACTGRAPH, how its features can support them in accessing this information, and observations regarding the coverage impact of test cases. We infer ten actionable recommendations on how developer tests can be visualized to help developers understand their behavior and impact.</p

Mutation Testing for Physical Computing

Physical computing, which builds interactive systems between the physical world and computers, has been widely used in a wide variety of domains and applications, e.g., the Internet of Things (IoT). Although physical computing has witnessed enormous realisations, testing these physical computing systems still face many challenges, such as potential circuit related bugs which are not part of the software problems, the timing issue which decreasing the testability, etc.; therefore, we proposed a mutation testing approach for physical computing systems to enable engineers to judge the quality of their tests in a more accurate way. The main focus is the communication between the software and peripherals. More particular, we first defined a set of mutation operators based on the common communication errors between the software and peripherals that could happen in the software. We conducted a preliminary experiment on nine physical computing projects based on the Raspberry Pi and Arduino platforms. The results show that our mutation testing method can assess the test suite quality effectively in terms of weakness and inadequacy.Accepted author manuscriptSoftware Engineerin

State-of-the-practice in quality assurance in Java-based open source software development

To ensure the quality of software systems, software engineers can make use of a variety of quality assurance approaches, for example, software testing, modern code review, automated static analysis, and build automation. Each of these quality assurance practices have been studied in depth in isolation, but there is a clear knowledge gap when it comes to our understanding of how these approaches are being used in conjunction, or not. In our study, we broadly investigate whether and how these quality assurance approaches are being used in conjunction in the development of 1454 popular open source software projects on GitHub. Our study indicates that typically projects do not follow all quality assurance practices together with high intensity. In fact, we only observe weak correlation among some quality assurance practices. In general, our study provides a deeper understanding of how existing quality assurance approaches are currently being used in Java-based open source software development. Besides, we specifically zoom in on the more mature projects in our dataset, and generally we observe that more mature projects are more intense in their application of the quality assurance practices, with more focus on their ASAT usage, and code reviewing, but no strong change in their CI usage.Software EngineeringSoftware Technolog

Generating Understandable Unit Tests through End-to-End Test Scenario Carving

Automatic unit test generators such as EvoSuite are able to automatically generate unit test suites with high coverage. This removes the burden of writing unit tests from developers, but the generated tests are often difficult to understand for them. In this paper, we introduce the MicroTestCarver approach that generates unit tests starting from manual or scripted end-toend (E2E) tests. Using carved information from these E2E tests, we generate unit tests that have meaningful test scenarios and contain actual test data. When we apply our MicroTestCarver approach, we observe that 85% of the generated tests are executable. Through a user study involving 20 participants, we get indications that tests generated with MicroTestCarver are relatively easy to understand.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Software Engineerin

Aiding Software Developers to Maintain Developer Tests

Preprint of paper published in: 16th European Conference on Software Maintenance and Reengineering (CSMR), 27-30 March 2012; doi:10.1109/CSMR.2012.12 Unit and integration tests can be invaluable during software maintenance as they help to understand pieces of code, they help with quality assurance and they build up confidence amongst developers. Unfortunately then, previous research has shown that unit tests do not always co-evolve nicely with the production code, thus leaving the software vulnerable. This paper presents TestNForce, a tool that helps developers to identify the unit tests that need to be altered and executed after a code change, thereby reducing the effort needed to keep the unit tests in sync with the changes to the production code. In order to evaluate TestNForce, we perform a user study that evaluates the adequacy, usefulness and completeness of TestNForce.Software Computer TechnologyElectrical Engineering, Mathematics and Computer Scienc

System Load Characterization Using Low-Level Performance Measurements

The performance of a software system directly influences customer satisfaction. Self-adaptiveness can contribute to this customer satisfaction by (1) taking appropriate measures when the performance becomes critical, e.g., the system load is too high, or (2) scheduling intensive tasks when the load is low. We investigate how self-adaptive systems can use low-level system measurements to characterize the load on a system. Our approach uses a combination of statistics and association rule learning to perform the characterization. We evaluate our approach using two case studies: a large-scale industrial system and a widely used synthetic benchmark (RUBiS). From our case studies follows that our approach is capable of closely characterizing the load on a system and that it is successful in detecting performance anomalies as well.Software Computer TechnologyElectrical Engineering, Mathematics and Computer Scienc

First Impressions of Using Stack Overflow for Education in a Computer Science Bachelor Programme

Community Question Answering (CQA) platforms like Stack Over- flow enable gamified and moderated community-driven knowledge creation. We report on our experiences of introducing Stack Over- flow in our Computer Science BSc programme and explore whether these platforms can (1) reduce the workload of lecturers in answer- ing questions in large-scale classrooms, and (2) create a community of learners in the large-scale setting of ∼1600 students.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Software EngineeringComputer Science & Engineering-Teaching Tea

A framework for quality assessment of just-in-time requirements: The case of open source feature requests

Until now, quality assessment of requirements has focused on traditional up-front requirements. Contrasting these traditional requirements are just-in-time (JIT) requirements, which are by definition incomplete, not specific and might be ambiguous when initially specified, indicating a different notion of "correctness." We analyze how the assessment of JIT requirements quality should be performed based on the literature of traditional and JIT requirements. Based on that analysis, we have designed a quality framework for JIT requirements and instantiated it for feature requests in open source projects. We also indicate how the framework can be instantiated for other types of JIT requirements. We have performed an initial evaluation of our framework for feature requests with eight practitioners from the Dutch agile community, receiving overall positive feedback. Subsequently, we have used our framework to assess 550 feature requests originating from three Open Source Software systems (Netbeans, ArgoUML and Mylyn Tasks). In doing so, we obtain a view on the feature request quality for the three open source projects. The value of our framework is threefold: (1) it gives an overview of quality criteria that are applicable to feature requests (at creation time or JIT); (2) it serves as a structured basis for teams that need to assess the quality of their JIT requirements; and (3) it provides a way to get an insight into the quality of JIT requirements in existing projects.Software Engineerin

Guest editorial: Mining software repositories 2018

Software Engineerin