A framework for the definition of metrics for actor-dependency models

Actor-dependency models are a formalism aimed at providing intentional descriptions of processes as a network of dependency relationships among actors. This kind of models is currently widely used in the early phase of requirements engineering as well as in other contexts such as organizational analysis and business process reengineering. In this paper, we are interested in the definition of a framework for the formulation of metrics over these models. These metrics are used to analyse the models with respect to some properties that are interesting for the system being modelled, such as security, efficiency or accuracy. The metrics are defined in terms of the actors and dependencies of the model. We distinguish three different kinds of metrics that are formally defined, and then we apply the framework at two different layers of a meeting scheduler system.Postprint (published version

Modelling the Strategic Alignment of Software Requirements using Goal Graphs

This paper builds on existing Goal Oriented Requirements Engineering (GORE) research by presenting a methodology with a supporting tool for analysing and demonstrating the alignment between software requirements and business objectives. Current GORE methodologies can be used to relate business goals to software goals through goal abstraction in goal graphs. However, we argue that unless the extent of goal-goal contribution is quantified with verifiable metrics and confidence levels, goal graphs are not sufficient for demonstrating the strategic alignment of software requirements. We introduce our methodology using an example software project from Rolls-Royce. We conclude that our methodology can improve requirements by making the relationships to business problems explicit, thereby disambiguating a requirement's underlying purpose and value.Comment: v2 minor updates: 1) bitmap images replaced with vector, 2) reworded related work ref[6] for clarit

Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design

Deep generative models such as Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), Diffusion Models, and Transformers, have shown great promise in a variety of applications, including image and speech synthesis, natural language processing, and drug discovery. However, when applied to engineering design problems, evaluating the performance of these models can be challenging, as traditional statistical metrics based on likelihood may not fully capture the requirements of engineering applications. This paper doubles as a review and practical guide to evaluation metrics for deep generative models (DGMs) in engineering design. We first summarize the well-accepted `classic' evaluation metrics for deep generative models grounded in machine learning theory. Using case studies, we then highlight why these metrics seldom translate well to design problems but see frequent use due to the lack of established alternatives. Next, we curate a set of design-specific metrics which have been proposed across different research communities and can be used for evaluating deep generative models. These metrics focus on unique requirements in design and engineering, such as constraint satisfaction, functional performance, novelty, and conditioning. Throughout our discussion, we apply the metrics to models trained on simple-to-visualize 2-dimensional example problems. Finally, we evaluate four deep generative models on a bicycle frame design problem and structural topology generation problem. In particular, we showcase the use of proposed metrics to quantify performance target achievement, design novelty, and geometric constraints. We publicly release the code for the datasets, models, and metrics used throughout the paper at https://decode.mit.edu/projects/metrics/

Regulatory Compliance-oriented Impediments and Associated Effort Estimation Metrics in Requirements Engineering for Contractual Systems Engineering Projects

Large-scale contractual systems engineering projects often need to comply with a myriad of government regulations and standards as part of contractual fulfillment. A key activity in the requirements engineering (RE) process for such a project is to elicit appropriate requirements from the regulations and standards that apply to the target system. However, there are impediments in achieving compliance due to such factors as: the voluminous contract and its high-level specifications, large number of regulatory documents, and multiple domains of the system. Little empirical research has been conducted on developing a shared understanding of the compliance-oriented complexities involved in such projects, and identifying and developing RE support (such as processes, tools, metrics, and methods) to improve overall performance for compliance projects. Through three studies on an industrial RE project, we investigated a number of issues in RE concerning compliance, leading to the following novel results:(i) a meta-model that captures artefacts-types and their compliance-oriented inter-relationships that exist in RE for contractual systems engineering projects; (ii) discovery of key impediments to requirements-compliance due to: (a) contractual complexities (e.g., regulatory requirements specified non-contiguously with non-regulatory requirements in the contract at the ratio of 1:19), (b) complexities in regulatory documents (e.g., over 300 regulatory documents being relevant to the subject system), and (c) large and complex system (e.g., 40% of the contractual regulatory requirements are cross-cutting); (iii) a method for deriving base metrics for estimating the effort needed to do compliance work during RE and demonstrate how a set of derived metrics can be used to create an effort estimation model for such work; (iv) a framework for structuring diverse regulatory documents and requirements for global product developments. These results lay a foundation in RE research on compliance issues with anticipation for its impact in real-world projects and in RE research

Evaluation of mobile health education applications for health professionals and patients

Paper presented at 8th International conference on e-Health (EH 2016), 1-3 July 2016, Funchal, Madeira, Portugal. ABSTRACT Mobile applications for health education are commonly utilized to support patients and health professionals. A critical evaluation framework is required to ensure the usability and reliability of mobile health education applications in order to facilitate the saving of time and effort for the various user groups; thus, the aim of this paper is to describe a framework for evaluating mobile applications for health education. The intended outcome of this framework is to meet the needs and requirements of the different user categories and to improve the development of mobile health education applications with software engineering approaches, by creating new and more effective techniques to evaluate such software. This paper first highlights the importance of mobile health education apps, then explains the need to establish an evaluation framework for these apps. The paper provides a description of the evaluation framework, along with some specific evaluation metrics: an efficient hybrid of selected heuristic evaluation (HE) and usability evaluation (UE) factors to enable the determination of the usefulness and usability of health education mobile apps. Finally, an explanation of the initial results for the framework was obtained using a Medscape mobile app. The proposed framework - An Evaluation Framework for Mobile Health Education Apps – is a hybrid of five metrics selected from a larger set in heuristic and usability evaluation, filtered based on interviews from patients and health professionals. These five metrics correspond to specific facets of usability identified through a requirements analysis of typical users of mobile health apps. These metrics were decomposed into 21 specific questionnaire questions, which are available on request from first author

Analysis of requirements quality evolution

Proceedings of: 40th International Conference on Software Engineering in Gothenburg, Sweden, May 27 - June 03, 2018A fundamental aspect in the requirements engineering process is to know the quality of a specification, including how the quality evolves over time. This paper introduces an industrial approach for analysis of requirements quality evolution. The approach has been implemented in the System Quality Analyzer tool, exploits quality metrics for requirements correctness, consistency, and completeness, and is based on the storage of quality information in snapshots that are combined and displayed in charts. This can help practitioners to assess the progress and status of a requirements engineering process and to make decisions.The AMASS project (H2020-ECSEL grant agreement no 692474; Spain's MINECO ref. PCIN-2015-262) has funded this work

Towards a framework for improving goal-oriented requirement models quality

Goal-orientation is a widespread and useful approach to Requirements Engineering. However, quality assessment frameworks focused on goal-oriented processes are either limited or remain on the theoretical side. Requirements quality initiatives range from simple metrics applicable to requirements documents, to general-purpose quality frameworks that include syntactic, semantic and pragmatic concerns. In some recent works, we have proposed a metrics framework for goal-oriented models, but the approach did not cover the cycle of quality assessment. In this paper we present a semiotic-based quality assessment proposal built upon the i* framework and the SEQUAL proposal. We propose a simplification of SEQUAL which can be applied to i* models by defining semantic, pragmatic and social metrics. As a result, we obtain suites of metrics that can be applied to i* goal-oriented requirements models. This theoretical work is put into practice by using iStarML, a XML representation of i* models, over which XQuery sentences compute the proposed metrics.Peer ReviewedPostprint (published version

Exploring Knowledge Engineering Strategies in Designing and Modelling a Road Traffic Accident Management Domain

Formulating knowledge for use in AI Planning engines is currently something of an ad-hoc process, where the skills of knowledge engineers and the tools they use may significantly influence the quality of the resulting planning application. There is little in the way of guidelines or standard procedures, however, for knowledge engineers to use when formulating knowledge into planning domain languages such as PDDL. This paper seeks to investigate this process using as a case study a road traffic accident management domain. Managing road accidents requires systematic, sound planning and coordination of resources to improve outcomes for accident victims. We have derived a set of requirements in consultation with stakeholders for the resource coordination part of managing accidents. We evaluate two separate knowledge engineering strategies for encoding the resulting planning domain from the set of requirements: (a) the traditional method of PDDL experts and text editor, and (b) a leading planning GUI with built in UML modelling tools. These strategies are evaluated using process and product metrics, where the domain model (the product) was tested extensively with a range of planning engines. The results give insights into the strengths and weaknesses of the approaches, highlight lessons learned regarding knowledge encoding, and point to important lines of research for knowledge engineering for planning

Analysis of expert’s opinion on requirements patterns for software product families framework using GQM method

Software product line engineering (SPLE), provides an opportunity to improve reuse of software artifacts through domain engineering and application engineering processes. During the domain engineering process, reuse activities of the product line are well-planned and subsequently executed in the application engineering process. This paper presents an analysis of interview result with experts in requirements engineering (RE) and software development for validating requirements pattern for software product families (RP-SPF) framework. The interview was conducted using goal questions metrics (GQM) method to define a goal and formulate research questions for conducting the interview. During the interview, 6 experts compared RP-SPF approach (systematic) with ad hoc (conventional) approach of reuse and documentation of requirements in terms of suitability, efficiency, and effectiveness in SPLE. The experts also gave their feedback on the perception of the use of RP-SPF tool. The analysis of the interview result shows that RP-SPF approach is suitable in SPLE and more efficient and effective than ad hoc approach of reuse and documentation of requirements