What Am I Testing and Where? Comparing Testing Procedures based on Lightweight Requirements Annotations

[Context] The testing of software-intensive systems is performed in different test stages each having a large number of test cases. These test cases are commonly derived from requirements. Each test stages exhibits specific demands and constraints with respect to their degree of detail and what can be tested. Therefore, specific test suites are defined for each test stage. In this paper, the focus is on the domain of embedded systems, where, among others, typical test stages are Software- and Hardware-in-the-loop. [Objective] Monitoring and controlling which requirements are verified in which detail and in which test stage is a challenge for engineers. However, this information is necessary to assure a certain test coverage, to minimize redundant testing procedures, and to avoid inconsistencies between test stages. In addition, engineers are reluctant to state their requirements in terms of structured languages or models that would facilitate the relation of requirements to test executions. [Method] With our approach, we close the gap between requirements specifications and test executions. Previously, we have proposed a lightweight markup language for requirements which provides a set of annotations that can be applied to natural language requirements. The annotations are mapped to events and signals in test executions. As a result, meaningful insights from a set of test executions can be directly related to artifacts in the requirements specification. In this paper, we use the markup language to compare different test stages with one another. [Results] We annotate 443 natural language requirements of a driver assistance system with the means of our lightweight markup language. The annotations are then linked to 1300 test executions from a simulation environment and 53 test executions from test drives with human drivers. Based on the annotations, we are able to analyze how similar the test stages are and how well test stages and test cases are aligned with the requirements. Further, we highlight the general applicability of our approach through this extensive experimental evaluation. [Conclusion] With our approach, the results of several test levels are linked to the requirements and enable the evaluation of complex test executions. By this means, practitioners can easily evaluate how well a systems performs with regards to its specification and, additionally, can reason about the expressiveness of the applied test stage.TU Berlin, Open-Access-Mittel - 202

An experimental Study using ACSL and Frama-C to formulate and verify Low-Level Requirements from a DO-178C compliant Avionics Project

Safety critical avionics software is a natural application area for formal verification. This is reflected in the formal method's inclusion into the certification guideline DO-178C and its formal methods supplement DO-333. Airbus and Dassault-Aviation, for example, have conducted studies in using formal verification. A large German national research project, Verisoft XT, also examined the application of formal methods in the avionics domain. However, formal methods are not yet mainstream, and it is questionable if formal verification, especially formal deduction, can be integrated into the software development processes of a resource constrained small or medium enterprise (SME). ESG, a Munich based medium sized company, has conducted a small experimental study on the application of formal verification on a small portion of a real avionics project. The low level specification of a software function was formalized with ACSL, and the corresponding source code was partially verified using Frama-C and the WP plugin, with Alt-Ergo as automated prover. We established a couple of criteria which a method should meet to be fit for purpose for industrial use in SME, and evaluated these criteria with the experience gathered by using ACSL with Frama-C on a real world example. The paper reports on the results of this study but also highlights some issues regarding the method in general which, in our view, will typically arise when using the method in the domain of embedded real-time programming.Comment: In Proceedings F-IDE 2015, arXiv:1508.0338

Towards a flexible service integration through separation of business rules

Driven by dynamic market demands, enterprises are continuously exploring collaborations with others to add value to their services and seize new market opportunities. Achieving enterprise collaboration is facilitated by Enterprise Application Integration and Business-to-Business approaches that employ architectural paradigms like Service Oriented Architecture and incorporate technological advancements in networking and computing. However, flexibility remains a major challenge related to enterprise collaboration. How can changes in demands and opportunities be reflected in collaboration solutions with minimum time and effort and with maximum reuse of existing applications? This paper proposes an approach towards a more flexible integration of enterprise applications in the context of service mediation. We achieve this by combining goal-based, model-driven and serviceoriented approaches. In particular, we pay special attention to the separation of business rules from the business process of the integration solution. Specifying the requirements as goal models, we separate those parts which are more likely to evolve over time in terms of business rules. These business rules are then made executable by exposing them as Web services and incorporating them into the design of the business process.\ud Thus, should the business rules change, the business process remains unaffected. Finally, this paper also provides an evaluation of the flexibility of our solution in relation to the current work in business process flexibility research

A System for Deduction-based Formal Verification of Workflow-oriented Software Models

The work concerns formal verification of workflow-oriented software models using deductive approach. The formal correctness of a model's behaviour is considered. Manually building logical specifications, which are considered as a set of temporal logic formulas, seems to be the significant obstacle for an inexperienced user when applying the deductive approach. A system, and its architecture, for the deduction-based verification of workflow-oriented models is proposed. The process of inference is based on the semantic tableaux method which has some advantages when compared to traditional deduction strategies. The algorithm for an automatic generation of logical specifications is proposed. The generation procedure is based on the predefined workflow patterns for BPMN, which is a standard and dominant notation for the modeling of business processes. The main idea for the approach is to consider patterns, defined in terms of temporal logic,as a kind of (logical) primitives which enable the transformation of models to temporal logic formulas constituting a logical specification. Automation of the generation process is crucial for bridging the gap between intuitiveness of the deductive reasoning and the difficulty of its practical application in the case when logical specifications are built manually. This approach has gone some way towards supporting, hopefully enhancing our understanding of, the deduction-based formal verification of workflow-oriented models.Comment: International Journal of Applied Mathematics and Computer Scienc

Requirements traceability in model-driven development: Applying model and transformation conformance

The variety of design artifacts (models) produced in a model-driven design process results in an intricate relationship between requirements and the various models. This paper proposes a methodological framework that simplifies management of this relationship, which helps in assessing the quality of models, realizations and transformation specifications. Our framework is a basis for understanding requirements traceability in model-driven development, as well as for the design of tools that support requirements traceability in model-driven development processes. We propose a notion of conformance between application models which reduces the effort needed for assessment activities. We discuss how this notion of conformance can be integrated with model transformations