Correlating context-awareness and mutation analysis for pervasive computing systems

Proceedings of the International Conference on Quality Software, 2010, p. 151-160Pervasive computing systems often use middleware as a means to communicate with the changing environment. However, the interactions with the context-aware middleware as well as the interactions among applications sharing the same middleware may introduce faults that are difficult to reveal by existing testing techniques. Our previous work proposed the notion of context diversity as a metric to measure the degree of changes in test inputs for pervasive software. In this paper, we present a case study on how much context diversity for test cases relates to fault-based mutants in pervasive software. Our empirical results show that conventional mutation operators can generate sufficient candidate mutants to support test effectiveness evaluation of pervasive software, and test cases with higher context diversity values tend to have higher mean mutation scores. On the other hand, for test cases sharing the same context diversity, their mutation scores can vary significantly in terms of standard derivations © 2010 IEEE.published_or_final_versio

Operator-based and random mutant selection: Better together

Abstract—Mutation testing is a powerful methodology for evaluating the quality of a test suite. However, the methodology is also very costly, as the test suite may have to be executed for each mutant. Selective mutation testing is a well-studied technique to reduce this cost by selecting a subset of all mutants, which would otherwise have to be considered in their entirety. Two common approaches are operator-based mutant selection, which only generates mutants using a subset of mutation operators, and random mutant selection, which selects a subset of mutants generated using all mutation operators. While each of the two approaches provides some reduction in the number of mutants to execute, applying either of the two to medium-sized, real-world programs can still generate a huge number of mutants, which makes their execution too expensive. This paper presents eight random sampling strategies defined on top of operator-based mutant selection, and empirically validates that operator-based selection and random selection can be applied in tandem to further reduce the cost of mutation testing. The experimental results show that even sampling only 5 % of mutants generated by operator-based selection can still provide precise mutation testing results, while reducing the average mutation testing time to 6.54 % (i.e., on average less than 5 minutes for this study). I

Uncertainty-wise Test Case Generation and Minimization for Cyber-Physical Systems

Cyber-Physical Systems (CPSs) typically operate in highly indeterminateenvironmental conditions, which require the development of testing methods that must explicitly consider uncertainty in test design, test generation, and test optimization. Towards this direction, we propose a set of uncertainty-wise test case generation and test case minimizationstrategies that rely on test ready models explicitly specifying subjective uncertainty. We propose two test case generation strategies and four test case minimizationstrategies based on the Uncertainty Theory and multi-objectivesearch. These strategies include a novel methodology for designing and introducing indeterminacy sources in the environment during test execution and a novel set of uncertainty-wise test verdicts. We performed an extensive empirical study to select the bestalgorithm out of eight commonly used multi-objective search algorithms, for each of the four minimizationstrategies, with five use cases of two industrial CPS case studies. The minimizedset of test cases obtained with the best algorithm for each minimizationstrategy were executedon the two real CPSs. The results showed that our best test strategy managed to observe 51% more uncertainties due to unknown indeterminate behaviorsof the physical environmentsof the CPSs as compared to the other test strategies. Also, the same test strategy managed to observe 118% more unknown uncertainties as compared to the unique number of known uncertainties.submittedVersio

Pembentukan Data Uji Menggunakan Algoritma Optimisasi Koloni Semut dan Pendekatan Teknik Pengujian Kotak Abu-Abu

Pengujian perangkat lunak dapat menjadi cara yang efektif untuk memperbaiki kualitas serta ketahanan perangkat lunak. Secara umum pengujian perangkat lunak dapat dilakukan dengan teknik kotak putih (white box) dan kotak hitam (black box). Teknik pengujian kotak abu-abu (gray box) dikenal sebagai teknik penguijan yang menggunakan pendekatan teknik pengujian kotak putih dan kotak hitam. Teknikpengujian kotak abu-abu memperluas kriteria cakupan logika dari teknik pengujian kotak putih dan menemukan semua kemungkinannya dari model desain perangkat lunak sepertipada teknik pengujian kotak hitam. Teknik pengujian kotak abu-abu menggunakan data seperti: UML Diagram, Model Arsitektur perangkat lunak, ataupun Finite State Machine Diagram (State Model) untuk membentuk kasus pengujian. Selain itu pengujian perangkat lunak dapat dilakukan dengan teknik Soft Computing dan Hard Computing. Teknik Hard Computing sulit diimplementasikan pada problematika yang ada saat ini. Sehinga teknik Soft Computing dapat menjadi alternatif yang digunakan dalam pengujian perangkat lunak. Teknik Soft Computing merupakan teknik yang lebih berfokus pada menginterpretasikan perilaku sistem dari pada hasil presisi. Teknik Soft Computing biasanya berdasarkan teknik logika fuzzy, jaringan saraf tiruan ataupun pengambilan keputusan berdasarkan nilai distribusi kemungkinan. Teknik pengujian perangkat lunak yang mengunakanteknil Soft Computing denganmetode pengambilan sampel berdasarkan nilai kemungkinan dikenal dengan teknik pengujian statistika. Berdasarkan penelitian terkait, algoritma Ant Colony Optimization (ACO) atau optimisasi koloni semut merupakan algoritma yang digunakan pada teknik pengujian statistika untuk membentuk data uji dengan kemampuan yang lebih baik dibandingkan algoritma lain seperti: Simulated Annealing (SA) serta algoritma genetika. Selain itu ACO juga memiliki hasil yang sebanding dengan algoritma Particle Swarm Optimization (PSO) atau optimisasi kawanan partikel. ACO diimplementasikan pada kode program perangkat lunak yang diuji untuk membentuk data uji berdasarkan nilai kemungkinan terbesar random data uji dari domain terpilih. Pemilihan data uji merupakan faktor utama yang menentukan keberhasilan dari suatu pengujian perangkat lunak. Sehingga pemilihan teknik yang tepat dapat membantu menunjang keberhasilan dalam pengujian perangkat lunak.Pada penelitian ini, ACO diimplementasikan berdasarkan teknik pengujian kotak abu-abu menggunakan diagram UML State Machine. Pembentukan data uji yang berkualitas adalah berdasarkan kecukupan kriteria percabangan yang dapat ditelusuri. Tujuan dari penelitian ini adalah untuk mendapatkan hasil perbandingan pembentukan data uji dengan teknik pengujian kotak abu-abu menggunakan diagram UML State Machine dan teknik pengujian struktur kotak putih menggunakan kode program. Hasil penelitian ini diharapkan mampu memberikan gambaran kualitas data uji yang dibentuk dari masing-masing teknik ========================================================================================================Software testing can be an effective way to improve software quality and reliability. In general, software testing techniques can be classified into White Box and Black Box. Gray Box testing technique is known as testing technique which is used both White Box and Black Box techniques. It extends the logical coverage criteria of white box method and finds all the possibility from the design model which is like black box method. Gray Box technique uses data such as: UML Diagram, Software Architecture Model, or Finite State Machine Diagram (State Model) to generate test cases. The other classification of software testing technique is Hard Computing technique and Soft Computing technique. Hard Computing technique is difficult to be implemented in today problems. And Soft Computing technique can be an alternative way which can be used in software testing. Soft Computing technique focuses on system behavior interpretation than precision result. Soft Computing technique is based on fuzzy logic, neural network, or decision making by probability distribution. Software testing which is used Soft Computing technique and its sampling method based on probability distribution, is known as statistical testing. Based on research, Ant Colony Optimization (ACO) Algorithm is algorithm which is used in statistical testing to generate test data, and its result better than another algorithm such as: Simulated Annealing (SA), and Genetic algorithm. Besides, its result is comparable with Paricle Swarm Optimization (PSO) algorithm. ACO was implemented in program code of software under test to generate test data based on the highest probability value of random test data from domain chosen. Test data selection is main factor which determines the software testing success. In this research, ACO was implemented based on Gray Box testing using UML State Machine Diagram. The quality of test data generated is based on branch coverage criteria. This research aims to get comparison result between Gray Box testing using UML State Machine Diagram and structural White Box Testing using program code in generating test data. The result of this research is expected to give description about the quality of test data generated from each technique

Identifying Defects Related to the Order in which Messages are Received in Message-Passing Systems

Improving the quality of software artifacts and products is an essential activity for everyone working on the development of software. Testing is one approach to reveal defects and faults in software. In recent years, message-passing systems have grown to a significant degree due to the rise of distributed systems, embedded systems, and so forth. In message-passing systems, components communicate with each other through sending and receiving messages. This message-passing mechanism introduces new opportunities for testing programs due to the fact that the time a message is delivered is not guaranteed, so the order in which messages are delivered is also not guaranteed. This non-determinism introduces interleaving and parallelization and subsequently a new source of software defects like race conditions. In this thesis, we have explained a new approach to testing a given component for identifying software faults related to the order in which messages are received by that component. We reorder messages coming to a certain component and deliver them in a different distinct ordering each time. We have three different methods for achieving message reordering: Blocking, Buffering, and Adaptive Buffering. We evaluate the effectiveness of our new testing methods using four metrics: Ordering Coverage, Coverage Rate, Slowdown Overhead, and Memory Overhead. We have implemented our Reordering Framework on QNX Neutrino 6.5.0 and compared our reordering methods with each other and with the naive random case using our experiments. We have also showed that our testing approach applies to real programs and can reveal real bugs in software