An Approach for the Comparative Evaluation of Requirements Formalisation Approaches

Abstract

Various approaches have been proposed to automate the formalisation of software requirements from semi-formal or informal documents. However, this area of research lacks well-established case studies to serve as benchmarks for comparing different methods. Additionally, there is a need for clear, objective criteria to effectively assess the outcomes of these formalisation approaches. These gaps make it challenging to identify which techniques are most suitable for specific formalisation tasks. This paper addresses these issues by introducing a set of standardized case studies and a structured framework for evaluating the performance of requirements formalisation techniques using measurable criteria. We apply this evaluation framework to assess five different formalisation methods, which include both rule-based and machine learning-driven approaches