Formal Modelling, Testing and Verification of HSA Memory Models using Event-B

The HSA Foundation has produced the HSA Platform System Architecture Specification that goes a long way towards addressing the need for a clear and consistent method for specifying weakly consistent memory. HSA is specified in a natural language which makes it open to multiple ambiguous interpretations and could render bugs in implementations of it in hardware and software. In this paper we present a formal model of HSA which can be used in the development and verification of both concurrent software applications as well as in the development and verification of the HSA-compliant platform itself. We use the Event-B language to build a provably correct hierarchy of models from the most abstract to a detailed refinement of HSA close to implementation level. Our memory models are general in that they represent an arbitrary number of masters, programs and instruction interleavings. We reason about such general models using refinements. Using Rodin tool we are able to model and verify an entire hierarchy of models using proofs to establish that each refinement is correct. We define an automated validation method that allows us to test baseline compliance of the model against a suite of published HSA litmus tests. Once we complete model validation we develop a coverage driven method to extract a richer set of tests from the Event-B model and a user specified coverage model. These tests are used for extensive regression testing of hardware and software systems. Our method of refinement based formal modelling, baseline compliance testing of the model and coverage driven test extraction using the single language of Event-B is a new way to address a key challenge facing the design and verification of multi-core systems.Comment: 9 pages, 10 figure

Distributed systems : architecture-driven specification using extended LOTOS

The thesis uses the LOTOS language (ISO International Standard ISO 8807) as a basis for the formal specification of distributed systems. Contributions are made to two key research areas: architecture-driven specification and LOTOS language extensions. The notion of architecture-driven specification is to guide the specification process by providing a reference-base of pre-defined domain-specific components. The thesis builds an infra-structure of architectural elements, and provides Extended LOTOS (XL) definitions of these elements. The thesis develops Extended LOTOS (XI.) for the specification of distributed systems. XL- is LOTOS enhanced with features for the formal specification of quantitative timing. probabilistic and priority requirements. For distributed systems, the specification of these ‘performance’ requirements, ran be as important as the specification of the associated functional requirements. To support quantitative timing features, the XL semantics define a global, discrete clock which can be used both to force events to occur at specific times, and to measure Intervals between event occurrences. XL introduces time policy operators ASAP (as soon as possible’ corresponding to “maximal progress semantics") and ALAP (late as possible'). Special internal transitions are introduced in XL semantics for the specification of probability, Conformance relations based on a notion of probabilization, together with a testing framework, are defined to support reasoning about probabilistic XL specifications. Priority within the XL semantics ensures that permitted events with the highest priority weighting of their class are allowed first. Both functional and performance specification play important roles in CIM (Computer Integrated Manufacturing) systems. The thesis uses a CIM system known as the CIM- OSA lntegrating Infrastructure as a case study of architecture-driven specification using XL. The thesis thus constitutes a step in the evolution of distributed system specification methods that have both an architectural basis and a formal basis

An Entry Point for Formal Methods: Specification and Analysis of Event Logs

Formal specification languages have long languished, due to the grave scalability problems faced by complete verification methods. Runtime verification promises to use formal specifications to automate part of the more scalable art of testing, but has not been widely applied to real systems, and often falters due to the cost and complexity of instrumentation for online monitoring. In this paper we discuss work in progress to apply an event-based specification system to the logging mechanism of the Mars Science Laboratory mission at JPL. By focusing on log analysis, we exploit the "instrumentation" already implemented and required for communicating with the spacecraft. We argue that this work both shows a practical method for using formal specifications in testing and opens interesting research avenues, including a challenging specification learning problem

Controllability problems in MSC-based testing

This is a pre-copyedited, author-produced PDF of an article accepted for publication in The Computer Journal following peer review. The definitive publisher-authenticated version [Dan, H and Hierons, RM (2012), "Controllability Problems in MSC-Based Testing", The Computer Journal, 55(11), 1270-1287] is available online at: http://comjnl.oxfordjournals.org/content/55/11/1270. Copyright @ The Authors 2011.In testing systems with distributed interfaces/ports, we may place a separate tester at each port. It is known that this approach can introduce controllability problems which have received much attention in testing from finite state machines. Message sequence charts (MSCs) form an alternative, commonly used, language for modelling distributed systems. However, controllability problems in testing from MSCs have not been thoroughly investigated. In this paper, controllability problems in MSC test cases are analysed with three notions of observability: local, tester and global. We identify two types of controllability problem in MSC-based testing. It transpires that each type of controllability problem is related to a type of MSC pathology. Controllability problems of timing are caused by races but not every race causes controllability problems; controllability problems of choice are caused by non-local choices and not every non-local choice causes controllability problems. We show that some controllability problems of timing are avoidable and some controllability problems of choice can be overcome when testers have better observational power. Algorithms are provided to tackle both types of controllability problems. Finally, we show how one can overcome controllability problems using a coordination service with status messages based on algorithms developed in this paper.EPSR

Towards more accurate real time testing

The languages Message Sequence Charts (MSC) [1], System Design Language1 (SDL) [2] and Testing and Test Control Notation Testing2 (TTCN-3) [3] have been developed for the design, modelling and testing of complex software systems. These languages have been developed to complement one another in the software development process. Each of these languages has features for describing, analysing or testing the real time properties of systems. Robust toolsets exist which provide integrated environments for the design, analysis and testing of systems, and it is claimed, for the complete development of real time systems. It was shown in [4] however, that there are fundamental problems with the SDL language and its associated tools for modelling and reasoning about real time systems. In this paper we present the limitations of TTCN-3 and propose recommendations which help minimise the timing inaccuracies that would otherwise occur in using the language directly

A framework for pathologies of message sequence charts

This is the post-print version of the final paper published in Information Software and Technology. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2012 Elsevier B.V.Context - It is known that a Message Sequence Chart (MSC) specification can contain different types of pathology. However, definitions of different types of pathology and the problems caused by pathologies are unclear, let alone the relationships between them. In this circumstance, it can be problematic for software engineers to accurately predict the possible problems that may exist in implementations of MSC specifications and to trace back to the design problems in MSC specifications from the observed problems of an implementation. Objective - We focus on generating a clearer view on MSC pathologies and building formal relationships between pathologies and the problems that they may cause. Method - By concentrating on the problems caused by pathologies, a categorisation of problems that a distributed system may suffer is first introduced. We investigate the different types of problems and map them to categories of pathologies. Thus, existing concepts related to pathology are refined and necessary concepts in the pathology framework are identified. Finally, we formally prove the relationships between the concepts in the framework. Results - A pathology framework is established as desired based on a restriction that considers problematic scenarios with a single undesirable event. In this framework, we define disjoint categories of both pathologies and the problems caused; the identified types of pathology are successfully mapped to the problems that they may cause. Conclusion - The framework achieved in this paper introduces taxonomies into and clarifies relationships between concepts in research on MSC pathologies. The taxonomies and relationships in the framework can help software engineers to predict problems and verify MSC specifications. The single undesirable event restriction not only enables a categorisation of pathological scenarios, but also has the potential practical benefit that a software engineer can concentrate on key problematic scenarios. This may make it easier to either remove pathologies from an MSC specification MM or test an implementation developed from MM for potential problems resulting from such pathologies

The Oracle Problem When Testing from MSCs

Message Sequence Charts (MSCs) form a popular language in which scenario-based specifications and models can be written. There has been significant interest in automating aspects of testing from MSCs. This paper concerns the Oracle Problem, in which we have an observation made in testing and wish to know whether this is consistent with the specification. We assume that there is an MSC specification and consider the case where we have entirely independent local testers (local observability) and where the observations of the local testers are logged and brought together (tester observability). It transpires that under local observability the Oracle Problem can be solved in low-order polynomial time if we use sequencing, loops and choices but becomes NP-complete if we also allow parallel components; if we place a bound on the number of parallel components then it again can be solved in polynomial time. For tester observability, the problem is NP-complete when we have either loops or choices. However, it can be solved in low-order polynomial time if we have only one loop, no choices, and no parallel components. If we allow parallel components then the Oracle Problem is NP-complete for tester observability even if we restrict to the case where there are at most two processes

What Am I Testing and Where? Comparing Testing Procedures based on Lightweight Requirements Annotations

[Context] The testing of software-intensive systems is performed in different test stages each having a large number of test cases. These test cases are commonly derived from requirements. Each test stages exhibits specific demands and constraints with respect to their degree of detail and what can be tested. Therefore, specific test suites are defined for each test stage. In this paper, the focus is on the domain of embedded systems, where, among others, typical test stages are Software- and Hardware-in-the-loop. [Objective] Monitoring and controlling which requirements are verified in which detail and in which test stage is a challenge for engineers. However, this information is necessary to assure a certain test coverage, to minimize redundant testing procedures, and to avoid inconsistencies between test stages. In addition, engineers are reluctant to state their requirements in terms of structured languages or models that would facilitate the relation of requirements to test executions. [Method] With our approach, we close the gap between requirements specifications and test executions. Previously, we have proposed a lightweight markup language for requirements which provides a set of annotations that can be applied to natural language requirements. The annotations are mapped to events and signals in test executions. As a result, meaningful insights from a set of test executions can be directly related to artifacts in the requirements specification. In this paper, we use the markup language to compare different test stages with one another. [Results] We annotate 443 natural language requirements of a driver assistance system with the means of our lightweight markup language. The annotations are then linked to 1300 test executions from a simulation environment and 53 test executions from test drives with human drivers. Based on the annotations, we are able to analyze how similar the test stages are and how well test stages and test cases are aligned with the requirements. Further, we highlight the general applicability of our approach through this extensive experimental evaluation. [Conclusion] With our approach, the results of several test levels are linked to the requirements and enable the evaluation of complex test executions. By this means, practitioners can easily evaluate how well a systems performs with regards to its specification and, additionally, can reason about the expressiveness of the applied test stage.TU Berlin, Open-Access-Mittel - 202

COST Action IC 1402 ArVI: Runtime Verification Beyond Monitoring -- Activity Report of Working Group 1

This report presents the activities of the first working group of the COST Action ArVI, Runtime Verification beyond Monitoring. The report aims to provide an overview of some of the major core aspects involved in Runtime Verification. Runtime Verification is the field of research dedicated to the analysis of system executions. It is often seen as a discipline that studies how a system run satisfies or violates correctness properties. The report exposes a taxonomy of Runtime Verification (RV) presenting the terminology involved with the main concepts of the field. The report also develops the concept of instrumentation, the various ways to instrument systems, and the fundamental role of instrumentation in designing an RV framework. We also discuss how RV interplays with other verification techniques such as model-checking, deductive verification, model learning, testing, and runtime assertion checking. Finally, we propose challenges in monitoring quantitative and statistical data beyond detecting property violation