Input Modeling Prioritization Using Statistically User Profile for Pairwise Test Case Generation with Constraints Handling

Pairwise testing is a widely used technique for software testing with reduce size of the test suite and able to detect interactions that trigger the system’s faults. In addition, pairwise testing test suites must be able to deal with constraints between input parameters and values. In current practice, selecting input parameters and values usually depends on tester skills that might not be sufficient. Input parameters and values modeling and tools for easily guiding and prioritizing the selection of optimal input parameters and values for the SUT is also required. In this work, we present an approach for prioritizing input parameters and values modeling using statistical user profile. Our approach is implemented in a tool called UPPTCT which provides ability to handle constraints on input parameters and values for pairwise testing in order to generate test cases. We conduct experiments to evaluate test case effectiveness and compare our tool with other renowned pairwise test generation and constraints handling tools. The experimental results show that the effectiveness of our approach is significantly more efficient and effective than random testing as large portion of reported defects with regard to statically user profile were caught by our approach. Furthermore, our tool performs better in some cases and performs comparable results for generating test cases upon input parameters and values for both with constraints handling and without constraints handling.Pairwise testing is a widely used technique for software testing with the reduced size of the test suite and able to detect interactions that trigger the system’s faults. In addition, pairwise testing test suites must be able to take constraints between input parameters and parameter values into account. In current practice, identifying and selecting input parameters and parameter values usually depends on tester skills that might not be sufficient. Input parameters and parameter values modeling and tools for easily guiding and prioritizing the selection of optimal input parameters and parameter values for the SUT is also required. In this work, we present an approach for prioritizing input parameters and parameter values modeling using statistical user profile. Our approach is implemented in a tool called UPPTCT which provides the ability to handle constraints on input parameters and parameter values for pairwise testing in order to generate test cases. We conduct experiments to evaluate test case effectiveness and compare our tool with other renowned pairwise test generation and constraints handling tools. The experimental results show that the effectiveness of our approach is significantly more efficient and effective than random testing as a large portion of reported defects with regard to statically user profile were caught by our approach. Furthermore, our tool performs better in some cases and performs comparable results for generating test cases upon input parameters and parameter values for both with constraints handling and without constraints handling

Automated pairwise testing approach based on classification tree modeling and negative selection algorithm

Generating the test cases for analysis is an important activity in software testing to increase the trust level of users. The traditional way to generate test cases is called exhaustive testing. It is infeasible and time consuming because it generates too many numbers of test cases. A combinatorial testing was used to solve the exhaustive testing problem. The popular technique in combinatorial testing is called pairwise testing that involves the interaction of two parameters. Although pairwise testing can cover the exhaustive testing problems, there are several issues that should be considered. First issue is related to modeling of the system under test (SUT) as a preprocess for test case generation as it has yet to be implemented in automated proposed approaches. The second issue is different approaches generate different number of test cases for different covering arrays. These issues showed that there is no one efficient way to find the optimal solution in pairwise testing that would consider the invalid combination or constraint. Therefore, a combination of Classification Tree Method and Negative Selection Algorithm (CTM-NSA) was developed in this research. The CTM approach was revised and enhanced to be used as the automated modeling and NSA approach was developed to optimize the pairwise testing by generate the low number of test cases. The findings showed that the CTM-NSA outperformed the other modeling method in terms of easing the tester and generating a low number of test cases in the small SUT size. Furthermore, it is comparable to the efficient approaches as compared to many of the test case generation approaches in large SUT size as it has good characteristic in detecting the self and non-self-sample. This characteristic occurs during the detection stage of NSA by covering the best combination of values for all parameters and considers the invalid combinations or constraints in order to achieve a hundred percent pairwise testing coverage. In addition, validation of the approach was performed using Statistical Wilcoxon Signed-Rank Test. Based on these findings, CTM-NSA had been shown to be able perform modeling in an automated way and achieve the minimum or a low number of test cases in small SUT size

A Survey of Constrained Combinatorial Testing

Combinatorial Testing (CT) is a potentially powerful testing technique, whereas its failure revealing ability might be dramatically reduced if it fails to handle constraints in an adequate and efficient manner. To ensure the wider applicability of CT in the presence of constrained problem domains, large and diverse efforts have been invested towards the techniques and applications of constrained combinatorial testing. In this paper, we provide a comprehensive survey of representations, influences, and techniques that pertain to constraints in CT, covering 129 papers published between 1987 and 2018. This survey not only categorises the various constraint handling techniques, but also reviews comparatively less well-studied, yet potentially important, constraint identification and maintenance techniques. Since real-world programs are usually constrained, this survey can be of interest to researchers and practitioners who are looking to use and study constrained combinatorial testing techniques

Methodologies for Evaluating User Centric Performance of Mobile Network Applications

Performance is an important attribute of mobile software applications, having a direct impact on end-user's experience. One of the obstacles that make software performance testing difficult to pursue is the lack of performance requirements that complicates the process of verifying the correctness of the test case output. Moreover, compared to other platforms, mobile applications' quality assurance is more challenging, since their functionality is affected by the surrounding environment. In this work, we propose methodologies and frameworks to evaluate the impact of interaction of the quality of the wireless network connection and application configurations on performance behaviour and performance robustness of a mobile networked application as perceived by the end user. We follow a model-based approach. The thesis starts by defining the system model of software applications that we target, the network stack that the application is assumed to use to provide the service to the end user, and the metric used to capture the quality of the provided network service. Then, an analytical performance model that captures the application-network interactions is developed using the Markovian framework. To model realistic interactions with the network, the performance model is developed and solved using supplementary variable technique (SVT). The model is intensively verified with simulation. Furthermore, two input network models are analytically developed. In both models, the mobile application is assumed to have a wireless network access through a WiFi access point that implements IEEE 802.11 protocol. In the first model, data transfer is achieved using user datagram protocol (UDP), while in the second, data transfer is accomplished using transmission control protocol (TCP). For the TCP model, two scenarios are considered. In the first scenario, an application data unit (APDU) is assumed to fit in one TCP packet, while in the second scenario, an APDU is assumed to fit in multiple TCP packets. All models are verified using the well-known NS2 network simulator. Third, we propose a model based test generation methodology to evaluate the impact of the interaction of the environment, the wireless network, and the application configurations on the performance of a mobile networked application. The methodology requires four artefacts as inputs, namely, a behaviour model of the software under test, a network model, a test coverage criterion, and a set of desired performance levels. The methodology consists of three steps: performance model development, test generation, and estimation of test execution parameters. To evaluate the end-user quality of experience, test generation is formulated as an inversion problem and solved as an optimization problem. To generate an efficient set of test cases, two test coverage criteria are proposed: user experience (UE) and user experience and input interaction (UEII). Test execution optimizations are inferred using a performance simulation model. To show the applicability of the methodology, two mobile networked app examples are used: multimedia streaming and web browsing. The effectiveness of the methodology is evaluated by comparing the time cost to design a test suite with random testing. The obtained results are very promising. Fourth, to minimize the incurred cost of performance model evaluations, we utilize metamorphic testing to generate test cases. Metamorphic testing is a technique that is proposed to alleviate the test oracle problem. By utilizing certain inherent properties of the system under test (metamorphic relations), test cases are generated and verified without the need to know the expected output of each individual test case in advance. By hybridizing our proposed test generation methodology with metamorphic testing, the time cost of generating a test suite is reduced tremendously. We first generate a limited set of seed test cases using our test generation methodology. Then, we generate a set of follow-up test cases by utilizing the developed network models as metamorphic relations and without the need to invoke the performance model. Follow-up test generation is formulated as a maximization problem. The objective is to maximize the distance between a seed test case and follow-up test cases so that to generate a non-redundant set of test cases. Three distance metrics are used: Euclidean, squared Euclidean, and Manhattan. The modified methodology is used to generate test cases for a multimedia streaming application. We empirically evaluate the modified test generation methodology using two evaluation metrics: the incurred time cost and the percentage of redundancy in the generated test suite. The obtained results show the advantage of the modified methodology in minimizing the cost of test generation process. Fifth, we propose a third methodology to evaluate the impact of the wireless network conditions on robustness of performance of adaptive and non-adaptive mobile networked applications. Software robustness is mainly about how the system behaves under stressful conditions. In this work, we target performance robustness under stressful network conditions. The proposed methodology consists of three steps and it requires three different artefacts as inputs. To quantify robustness, two metrics (static and dynamic robustness) are proposed. The main challenge in evaluating robustness is the combinatorial growth of network-application interactions that need to be evaluated. To mitigate this issue, we propose an algorithm to limit the number of interactions, utilizing the monotonicity property of the performance model. To evaluate the dynamic robustness metric, the ability of the adaptive application to tolerate degraded network conditions has to be evaluated. This problem is formulated as a minimization problem. The methodology is used to evaluate the performance robustness of a mobile multimedia streaming application. The effectiveness of the proposed methodology is evaluated. The obtained results show three to five times reduction in total cost compared to the naive approach in which all combinations are exhaustively evaluated