Finding a suitable performance testing tool

Abstract. The pursuit of finding the most suitable testing software for each project is a difficult task as there are a lot of software effective finding certain kind of problems but completely missing others in the field of stress and load testing. A silver bullet solving all problems in a cost effective and reliable way has not yet been found. This project was done as a systematic literature review to find whether there are solutions documented capable of testing everything in a cost-effective way. The document starts with an introduction of the task, originating from a real software testing company’s suggestion of finding suitable test software that can, cost effectively and reliably, fulfil the needs of the company. A history section is describing the reason of testing importance, basics of testing and what others have found in their studies of the area. The research method is described in detail followed by results describing tools found during the research divided in sections by license type. The sectioning by license type was selected for the benefit of testing companies that are interested in further developing tools found to their own interest. Findings and answered research questions were presented and discussed followed by possible implications and further research suggestions to future scholars interested in the matter. The systematic literature review found a total of 40 different tools identified during the data extraction process. One complete software system was available commercially including heavy support and help functions for the customer. A different approach linking open source and relatively inexpensive pieces of software together to achieve a composite solution was also identified. The solution included the most common and most popular individual piece of software identified by the study. All found pieces of software were listed and commented briefly mainly with information originating from the authors’ home pages

Turku Centre for Computer Science – Annual Report 2013

Due to a major reform of organization and responsibilities of TUCS, its role, activities, and even structures have been under reconsideration in 2013. The traditional pillar of collaboration at TUCS, doctoral training, was reorganized due to changes at both universities according to the renewed national system for doctoral education. Computer Science and Engineering and Information Systems Science are now accompanied by Mathematics and Statistics in newly established doctoral programs at both University of Turku and &Aring;bo Akademi University. Moreover, both universities granted sufficient resources to their respective programmes for doctoral training in these fields, so that joint activities at TUCS can continue. The outcome of this reorganization has the potential of proving out to be a success in terms of scientific profile as well as the quality and quantity of scientific and educational results.&nbsp; International activities that have been characteristic to TUCS since its inception continue strong. TUCS&rsquo; participation in European collaboration through EIT ICT Labs Master&rsquo;s and Doctoral School is now more active than ever. The new double degree programs at MSc and PhD level between University of Turku and Fudan University in Shaghai, P.R.China were succesfully set up and are&nbsp; now running for their first year. The joint students will add to the already international athmosphere of the ICT House.&nbsp; The four new thematic reseach programmes set up acccording to the decision by the TUCS Board have now established themselves, and a number of events and other activities saw the light in 2013. The TUCS Distinguished Lecture Series managed to gather a large audience with its several prominent speakers. The development of these and other research centre activities continue, and&nbsp; new practices and structures will be initiated to support the tradition of close academic collaboration.&nbsp; The TUCS&rsquo; slogan Where Academic Tradition Meets the Exciting Future has proven true throughout these changes. Despite of the dark clouds on the national and European economic sky, science and higher education in the field have managed to retain all the key ingredients for success. Indeed, the future of ICT and Mathematics in Turku seems exciting.</p

WESSBAS: extraction of probabilistic workload specifications for load testing and performance prediction—a model-driven approach for session-based application systems

The specification of workloads is required in order to evaluate performance characteristics of application systems using load testing and model-based performance prediction. Defining workload specifications that represent the real workload as accurately as possible is one of the biggest challenges in both areas. To overcome this challenge, this paper presents an approach that aims to automate the extraction and transformation of workload specifications for load testing and model-based performance prediction of session-based application systems. The approach (WESSBAS) comprises three main components. First, a system- and tool-agnostic domain-specific language (DSL) allows the layered modeling of workload specifications of session-based systems. Second, instances of this DSL are automatically extracted from recorded session logs of production systems. Third, these instances are transformed into executable workload specifications of load generation tools and model-based performance evaluation tools. We present transformations to the common load testing tool Apache JMeter and to the Palladio Component Model. Our approach is evaluated using the industry-standard benchmark SPECjEnterprise2010 and the World Cup 1998 access logs. Workload-specific characteristics (e.g., session lengths and arrival rates) and performance characteristics (e.g., response times and CPU utilizations) show that the extracted workloads match the measured workloads with high accuracy

Methodologies for Evaluating User Centric Performance of Mobile Network Applications

Performance is an important attribute of mobile software applications, having a direct impact on end-user's experience. One of the obstacles that make software performance testing difficult to pursue is the lack of performance requirements that complicates the process of verifying the correctness of the test case output. Moreover, compared to other platforms, mobile applications' quality assurance is more challenging, since their functionality is affected by the surrounding environment. In this work, we propose methodologies and frameworks to evaluate the impact of interaction of the quality of the wireless network connection and application configurations on performance behaviour and performance robustness of a mobile networked application as perceived by the end user. We follow a model-based approach. The thesis starts by defining the system model of software applications that we target, the network stack that the application is assumed to use to provide the service to the end user, and the metric used to capture the quality of the provided network service. Then, an analytical performance model that captures the application-network interactions is developed using the Markovian framework. To model realistic interactions with the network, the performance model is developed and solved using supplementary variable technique (SVT). The model is intensively verified with simulation. Furthermore, two input network models are analytically developed. In both models, the mobile application is assumed to have a wireless network access through a WiFi access point that implements IEEE 802.11 protocol. In the first model, data transfer is achieved using user datagram protocol (UDP), while in the second, data transfer is accomplished using transmission control protocol (TCP). For the TCP model, two scenarios are considered. In the first scenario, an application data unit (APDU) is assumed to fit in one TCP packet, while in the second scenario, an APDU is assumed to fit in multiple TCP packets. All models are verified using the well-known NS2 network simulator. Third, we propose a model based test generation methodology to evaluate the impact of the interaction of the environment, the wireless network, and the application configurations on the performance of a mobile networked application. The methodology requires four artefacts as inputs, namely, a behaviour model of the software under test, a network model, a test coverage criterion, and a set of desired performance levels. The methodology consists of three steps: performance model development, test generation, and estimation of test execution parameters. To evaluate the end-user quality of experience, test generation is formulated as an inversion problem and solved as an optimization problem. To generate an efficient set of test cases, two test coverage criteria are proposed: user experience (UE) and user experience and input interaction (UEII). Test execution optimizations are inferred using a performance simulation model. To show the applicability of the methodology, two mobile networked app examples are used: multimedia streaming and web browsing. The effectiveness of the methodology is evaluated by comparing the time cost to design a test suite with random testing. The obtained results are very promising. Fourth, to minimize the incurred cost of performance model evaluations, we utilize metamorphic testing to generate test cases. Metamorphic testing is a technique that is proposed to alleviate the test oracle problem. By utilizing certain inherent properties of the system under test (metamorphic relations), test cases are generated and verified without the need to know the expected output of each individual test case in advance. By hybridizing our proposed test generation methodology with metamorphic testing, the time cost of generating a test suite is reduced tremendously. We first generate a limited set of seed test cases using our test generation methodology. Then, we generate a set of follow-up test cases by utilizing the developed network models as metamorphic relations and without the need to invoke the performance model. Follow-up test generation is formulated as a maximization problem. The objective is to maximize the distance between a seed test case and follow-up test cases so that to generate a non-redundant set of test cases. Three distance metrics are used: Euclidean, squared Euclidean, and Manhattan. The modified methodology is used to generate test cases for a multimedia streaming application. We empirically evaluate the modified test generation methodology using two evaluation metrics: the incurred time cost and the percentage of redundancy in the generated test suite. The obtained results show the advantage of the modified methodology in minimizing the cost of test generation process. Fifth, we propose a third methodology to evaluate the impact of the wireless network conditions on robustness of performance of adaptive and non-adaptive mobile networked applications. Software robustness is mainly about how the system behaves under stressful conditions. In this work, we target performance robustness under stressful network conditions. The proposed methodology consists of three steps and it requires three different artefacts as inputs. To quantify robustness, two metrics (static and dynamic robustness) are proposed. The main challenge in evaluating robustness is the combinatorial growth of network-application interactions that need to be evaluated. To mitigate this issue, we propose an algorithm to limit the number of interactions, utilizing the monotonicity property of the performance model. To evaluate the dynamic robustness metric, the ability of the adaptive application to tolerate degraded network conditions has to be evaluated. This problem is formulated as a minimization problem. The methodology is used to evaluate the performance robustness of a mobile multimedia streaming application. The effectiveness of the proposed methodology is evaluated. The obtained results show three to five times reduction in total cost compared to the naive approach in which all combinations are exhaustively evaluated