Lessons learnt from using DSLs for automated software testing

Domain Specific Languages (DSLs) provide a means of unambiguously expressing concepts in a particular domain. Although they may not refer to it as such, companies build and maintain DSLs for software testing on a day-to-day basis, especially when they define test suites using the Gherkin language. However, although the practice of specifying and automating test cases using the Gherkin language and related technologies such as Cucumber has become mainstream, the curation of such languages presents a number of challenges. In this paper we discuss lessons learnt from five case studies on industry systems, two involving the use of Gherkin-type syntax and another three case studies using more rigidly defined language grammars. Initial observations indicate that the likelihood of success of such efforts is increased if one manages to use an approach which separates the concerns of domain experts who curate the language, users who write scripts with the language, and engineers who wire the language into test automation technologies thus producing executable test code. We also provide some insights into desirable qualities of testing DSLs in different contexts.peer-reviewe

Making Property-Based Testing Easier to Read for Humans

Software stakeholders who do not have a technical profile (i.e. users, clients) but do want to take part in the development and/or quality assurance process of software, have an unmet need for communication on what is being tested during the development life-cycle. The transformation of test properties and models into semi-natural language representations is one way of responding to such need. Our research has demonstrated that these transformations are challenging but feasible, and they have been implemented into a prototype tool called readSpec. The readSpec tool transforms universally-quantified test properties and stateful test models - the two kinds of test artifacts used in property-based testing - into plain text interpretations. The tool has been successfully evaluated on the PBT artifacts produced and used within the FP7 PROWESS project by industrial partners