Model-based Testing: Next Generation Functional Software Testing

The idea of model-based testing is to use an explicit abstract model of a SUT and its environment to automatically derive tests for the SUT: the behavior of the model of the SUT is interpreted as the intended behavior of the SUT. The technology of automated model-based test case generation has matured to the point where large-scale deployments of this technology are becoming commonplace. The prerequisites for success, such as qualification of the test team, integrated tool chain availability and methods, are now identified, and a wide range of commercial and open-source tools are available. Although MBT will not solve all testing problems, it is an important and useful technique, which brings significant progress over the state of the practice for functional software testing effectiveness, and can increase productivity and improve functional coverage. In this talk, we\u27ll adress the current trend of deploying MBT in the industry, particularly in the TCoE - Test Center of Excellence - managed by the big System Integrators, as a vector for software testing "industrialization"

A Review on Web Application Testing and its Current Research Directions

Testing is an important part of every software development process on which companies devote considerable time and effort. The burgeoning web applications and their proliferating economic significance in the society made the area of web application testing an area of acute importance. The web applications generally tend to take faster and quicker release cycles making their testing very challenging. The main issues in testing are cost efficiency and bug detection efficiency. Coverage-based   testing is the process of ensuring exercise of specific program elements. Coverage measurement helps determine the “thoroughness” of testing achieved. An avalanche of tools, techniques, frameworks came into existence to ascertain the quality of web applications.  A comparative study of some of the prominent tools, techniques and models for web application testing is presented. This work highlights the current research directions of some of the web application testing techniques

DOMAIN SPECIFIC TEST CASE GENERATION USING HIGHER ORDERED TYPED LANGUAGES FOR SPECIFICATION

Model based testing is an approach for automatic generation of test cases on the basis of a representative model of the system. Recent studies show that model based testing has many possible advantages over manual test generation techniques including a gain in effectiveness, efficiency and reuse. The effectiveness (ability to uncover faults in a system) of a model based testing process is determined by the correctness of the model and by the number of requirements represented in the model. In practice, test models for model based test automation techniques are usually created from requirement or design specifications of the software and hence, these techniques overtly rely on such specifications for the completeness of the test models. This may lead to failure in testing some critical requirements specific to the application domain because the user, who helps in defining the requirements, may fail to consider certain domain specific requirements. To him some may appear to be too trivial to be specified explicitly in the requirements document and the others, he may forget. Even if the requirement is complete with domain specific requirements, testers may not realize criticality of such requirements or may find them too complex to model. In all such cases, testing is incomplete and ineffective. This dissertation describes a new model based testing technique developed to remedy such situations. The new technique is based on modeling the system under test using a strongly typed domain specific language (DSL). In the new technique, information about domain specific requirements of an application are captured automatically by exploiting properties of the DSL and are subsequently introduced in the test model. The new technique is applied to generate test cases for the applications interfacing with relational databases and the example DSL chosen for that purpose is HaskellDB. Test suites generated using the new technique are enriched with test cases addressing domain specific implicit requirements and therefore, are more effective in finding faults. This dissertation will present details of the technique and describe an experiment and a case study to explore its effectiveness, efficiency, usability and industrial applicability

A Review on Tools, Mechanics, Benefits, and Challenges of Gamified Software Testing

Gamification is an established practice in Software Engineering to increase effectiveness and engagement in many practices. This manuscript provides a characterisation of the application of gamification to the Software Testing area. Such practice in fact reportedly suffers from low engagement by both personnel in industrial contexts and learners in educational contexts. Our goal is to identify the application areas and utilised gamified techniques and mechanics, the provided benefits and drawbacks, as well as the open challenges in the field. To this purpose, we conducted a Multivocal Literature Review to identify white and grey literature sources addressing gamified software testing. We analysed 73 contributions and summarised the most common gamified mechanics, concepts, tools and domains where they are mostly applied. We conclude that gamification in software testing is mostly applied to the test creation phase with simple white-box unit or mutation testing tools, and is mostly used to foster good behaviours by promoting the testers’ accomplishment. Key research areas and main challenges in the field are: careful design of tailored gamified mechanics for specific testing techniques; the need for technological improvements to enable crowdsourcing, cooperation, and concurrency; the necessity for empirical and large-scale evaluation of the benefits delivered by gamification mechanics

Defining and Evaluating Test Suite Consolidation for Event Sequence-based Test Cases

This research presents a new test suite consolidation technique, called CONTEST, for automated GUI testing. A new probabilistic model of the GUI is developed to allow direct application of CONTEST. Multiple existing test suites are used to populate the model and compute probabilities based on the observed event sequences. These probabilities are used to generate a new test suite that consolidates the original ones. A new test suite similarity metric, called CONTeSSi(n), is introduced which compares multiple event sequence-based test suites using relative event positions. Results of empirical studies showed that CONTEST yields a test suite that achieves better fault detection and code coverage than the original suites, and that the CONTeSSi(n) metric is a better indicator of the similarity between sequence-based test suites than existing metrics

On the Evolution of Keyword-Driven Test Suites

Many companies rely on software testing to verify that their software products meet their requirements. However, test quality and, in particular, the quality of end-to-end testing is relatively hard to achieve. The problem becomes challenging when software evolves, as end-to-end test suites need to adapt and conform to the evolved software. Unfortunately, end-to-end tests are particularly fragile as any change in the application interface, e.g., application flow, location or name of graphical user interface elements, necessitates a change in the tests. This paper presents an industrial case study on the evolution of Keyword-Driven test suites, also known as Keyword-Driven Testing (KDT). Our aim is to demonstrate the problem of test maintenance, identify the benefits of Keyword-Driven Testing and overall improve the understanding of test code evolution (at the acceptance testing level). This information will support the development of automatic techniques, such as test refactoring and repair, and will motivate future research. To this end, we identify, collect and analyze test code changes across the evolution of industrial KDT test suites for a period of eight months. We show that the problem of test maintenance is largely due to test fragility (most commonly-performed changes are due to locator and synchronization issues) and test clones (over 30% of keywords are duplicated). We also show that the better test design of KDT test suites has the potential for drastically reducing (approximately 70%) the number of test code changes required to support software evolution. To further validate our results, we interview testers from BGL BNP Paribas and report their perceptions on the advantages and challenges of keyword-driven testing

Object-Oriented Design of an Automated Calibration System for an Analog I/O Process Control Device

The goal of this project was to deliver the design of object-oriented software for the control of a custom calibration/test system. This calibration/test system is to be used for the production testing and calibration of an Analog Input device used in a process control system. Software features include a GUI (Graphical User Interface), Analog Input device troubleshooting tools, calibration/test system calibration and troubleshooting tools, and report printing capability. This project followed the methodology defined by the Unified Process Model and delivered design documentation through the Inception and Elaboration phases. This paper discusses the planning, documentation and testing differences between a large software project and a small software project

Automated test of evolving software

A thesis submitted to the University of Luton, in partial fulfilment of the requirements for the degree of Doctor of PhilosophyComputers and the software they run are pervasive, yet released software is often unreliable, which has many consequences. Loss of time and earnings can be caused by application software (such as word processors) behaving incorrectly or crashing. Serious disruption can occur as in the l4th August 2003 blackouts in North East USA and Canadal, or serious injury or death can be caused as in the Therac-25 overdose incidents. One way to improve the quality of software is to test it thoroughly. However, software testing is time consuming, the resources, capabilities and skills needed to carry it out are often not available and the time required is often curtailed because of pressures to meet delivery deadlines3. Automation should allow more thorough testing in the time available and improve the quality of delivered software, but there are some problems with automation that this research addresses. Firstly, it is difficult to determine ifthe system under test (SUT) has passed or failed a test. This is known as the oracle problem4 and is often ignored in software testing research. Secondly, many software development organisations use an iterative and incremental process, known as evolutionary development, to write software. Following release, software continues evolving as customers demand new features and improvements to existing ones5. This evolution means that automated test suites must be maintained throughout the life ofthe software. A contribution of this research is a methodology that addresses automatic generation of the test cases, execution of the test cases and evaluation of the outcomes from running each test. "Predecessor" software is used to solve the oracle problem. This is software that already exists, such as a previous version of evolving software, or software from a different vendor that solves the same, or similar, problems. However, the resulting oracle is assumed not be perfect, so rules are defined in an interface, which are used by the evaluator in the test evaluation stage to handle the expected differences. The interface also specifies functional inputs and outputs to the SUT. An algorithm has been developed that creates a Markov Chain Transition Matrix (MCTM) model of the SUT from the interface. Tests are then generated automatically by making a random walk of the MCTM. This means that instead of maintaining a large suite of tests, or a large model of the SUT, only the interface needs to be maintained. 1) NERC Steering Group (2004). Technical Analysis ofthe August 14,2003, Blackout: What Happened, Why, and What Did We Learn? July 13th 2004. Available from: ftp:/ /www.nerc.com/pub/sys/all_ updl/docslblackoutINERC ]inatBlackout_Report _ 07_13_ 04.pdf 2) Leveson N. G., Turner C. S. (1993) An investigation of the Therac-25 accidents. IEEE Computer, Vo126, No 7, Pages 18-41. 3) LogicaCMG (2005) Testing Times for Board Rooms. Available from http://www.logicacmg.com/pdf/trackeditestingTimesBoardRooms.pdf 4) Bertolino, A. (2003) Software Testing Research and Practice, ASM 2003, Lecture Notes in Computer Science, Vol 2589, Pages 1-21. 5) Sommerville, 1. (2004) Software Engineering, 7th Edition. Addison Wesley. ISBN 0-321-21026-3