Validating Behavioral Requirements, Conditions, and Rules of Autonomous Systems with Scenario-Based Testing

Assuring the safety of autonomous vehicles is more and more approached by using scenario-based testing. Relevant driving situations are utilized here to fuel the argument that an autonomous vehicle behaves correctly. Many recent works focus on the specification, variation, generation, and execution of individual scenarios. However, it is still an open question if operational design domains, which describe the environmental conditions under which the system under test has to function, can be assessed with scenario-based testing. In this paper, we present open challenges and resulting research questions in the field of assuring the safety of autonomous vehicles. We have developed a toolchain that enables us to conduct scenario-based testing experiments based on scenario classification with temporal logic and driving data obtained from the CARLA simulator. We discuss the toolchain and present first results using analysis metrics like class coverage or distribution

What Am I Testing and Where? Comparing Testing Procedures based on Lightweight Requirements Annotations

[Context] The testing of software-intensive systems is performed in different test stages each having a large number of test cases. These test cases are commonly derived from requirements. Each test stages exhibits specific demands and constraints with respect to their degree of detail and what can be tested. Therefore, specific test suites are defined for each test stage. In this paper, the focus is on the domain of embedded systems, where, among others, typical test stages are Software- and Hardware-in-the-loop. [Objective] Monitoring and controlling which requirements are verified in which detail and in which test stage is a challenge for engineers. However, this information is necessary to assure a certain test coverage, to minimize redundant testing procedures, and to avoid inconsistencies between test stages. In addition, engineers are reluctant to state their requirements in terms of structured languages or models that would facilitate the relation of requirements to test executions. [Method] With our approach, we close the gap between requirements specifications and test executions. Previously, we have proposed a lightweight markup language for requirements which provides a set of annotations that can be applied to natural language requirements. The annotations are mapped to events and signals in test executions. As a result, meaningful insights from a set of test executions can be directly related to artifacts in the requirements specification. In this paper, we use the markup language to compare different test stages with one another. [Results] We annotate 443 natural language requirements of a driver assistance system with the means of our lightweight markup language. The annotations are then linked to 1300 test executions from a simulation environment and 53 test executions from test drives with human drivers. Based on the annotations, we are able to analyze how similar the test stages are and how well test stages and test cases are aligned with the requirements. Further, we highlight the general applicability of our approach through this extensive experimental evaluation. [Conclusion] With our approach, the results of several test levels are linked to the requirements and enable the evaluation of complex test executions. By this means, practitioners can easily evaluate how well a systems performs with regards to its specification and, additionally, can reason about the expressiveness of the applied test stage.TU Berlin, Open-Access-Mittel - 202

The determination of measures of software reliability

Measurement of software reliability was carried out during the development of data base software for a multi-sensor tracking system. The failure ratio and failure rate were found to be consistent measures. Trend lines could be established from these measurements that provide good visualization of the progress on the job as a whole as well as on individual modules. Over one-half of the observed failures were due to factors associated with the individual run submission rather than with the code proper. Possible application of these findings for line management, project managers, functional management, and regulatory agencies is discussed. Steps for simplifying the measurement process and for use of these data in predicting operational software reliability are outlined

Challenges and Directions in Formalizing the Semantics of Modeling Languages

Developing software from models is a growing practice and there exist many model-based tools (e.g., editors, interpreters, debuggers, and simulators) for supporting model-driven engineering. Even though these tools facilitate the automation of software engineering tasks and activities, such tools are typically engineered manually. However, many of these tools have a common semantic foundation centered around an underlying modeling language, which would make it possible to automate their development if the modeling language specification were formalized. Even though there has been much work in formalizing programming languages, with many successful tools constructed using such formalisms, there has been little work in formalizing modeling languages for the purpose of automation. This paper discusses possible semantics-based approaches for the formalization of modeling languages and describes how this formalism may be used to automate the construction of modeling tools

Harmonizing CMMI-DEV 1.2 and XP Method to Improve The Software Development Processes in Small Software Development Firms

Most software development organizations are small firms, and they have realized the need to manage and improve their software development and management activities. Traditional Software Process Improvement (SPI) models and standards are not realistic for these firms because of high cost, limited resources and strict project deadlines. Therefore, these firms need a lightweight software development method and an appropriate SPI model to manage and improve their software development and management processes. This study aims to construct a suitable software development process improvement framework for Small Software Development Firms (SSDFs) based on eXtreme Programming (XP) method and Capability Maturity Model Integration for Development Version 1.2 (CMMI-Dev1.2) model. Four stages are involved in developing the framework: (1) aligning XP practices to the specific goals of CMMI-Dev1.2 Key Process Areas (KPAs); (2) developing the proposed software development process improvement framework based on extending XP method by adapting the Extension-Based Approach (EBA), CMMI-Dev1.2, and generic elements of the SPI framework; (3) verifying the compatibility of the proposed framework to the KPAs of CMMI-Dev1.2 by using focus group method coupled with Delphi technique; and (4) validating the modified framework by using CMMI-Dev1.2 questionnaire as a main item to validate the suitability of the modified framework for SSDFs, and conducting two case studies to validate the applicability and effectiveness of this framework for these firms. The result of aligning XP practices to the KPAs of CMMI-Dev1.2 shows that twelve KPAs are largely supported by XP practices, eight KPAs are partially supported by XP practices, and two KPAs are not-supported by XP practices. The main contributions of this study are: software development process improvement framework for SSDFs, elicit better understanding of how to construct the framework, and quality improvement of the software development processes. There are possible avenues for extending this research to fulfil the missing specific practices of several KPAs, examining other agile practices and using CMMI-Dev1.3 to improve the framework, and conducting more case studie

Ontology in Information Security

The past several years we have witnessed that information has become the most precious asset, while protection and security of information is becoming an ever greater challenge due to the large amount of knowledge necessary for organizations to successfully withstand external threats and attacks. This knowledge collected from the domain of information security can be formally described by security ontologies. A large number of researchers during the last decade have dealt with this issue, and in this paper we have tried to identify, analyze and systematize the relevant papers published in scientific journals indexed in selected scientific databases, in period from 2004 to 2014. This paper gives a review of literature in the field of information security ontology and identifies a total of 52 papers systematized in three groups: general security ontologies (12 papers), specific security ontologies (32 papers) and theoretical works (8 papers). The papers were of different quality and level of detail and varied from presentations of simple conceptual ideas to sophisticated frameworks based on ontology