Some Issues in the Testing of Computer Simulation Models

The testing of simulation models has much in common with testing processes in other types of application involving software development. However, there are also important differences associated with the fact that simulation model testing involves two distinct aspects, which are known as verification and validation. Model validation is concerned with investigation of modelling errors and model limitations while verification involves checking that the simulation program is an accurate representation of the mathematical and logical structure of the underlying model. Success in model validation depends upon the availability of detailed information about all aspects of the system being modelled. It also may depend on the availability of high quality data from the system which can be used to compare its behaviour with that of the corresponding simulation model. Transparency, high standards of documentation and good management of simulation models and data sets are basic requirements in simulation model testing. Unlike most other areas of software testing, model validation often has subjective elements, with potentially important contributions from face- validation procedures in which experts give a subjective assessment of the fidelity of the model. Verification and validation processes are not simply applied once but must be used repeatedly throughout the model development process, with regressive testing principles being applied. Decisions about when a model is acceptable for the intended application inevitably involve some form of risk assessment. A case study concerned with the development and application of a simulation model of a hydro-turbine and electrical generator system is used to illustrate some of the issues arising in a typical control engineering application. Results from the case study suggest that it is important to bring together objective aspects of simulation model testing and the more subjective face- validation aspects in a coherent fashion. Suggestions are also made about the need for changes in approach in the teaching of simulation techniques to engineering students to give more emphasis to issues of model quality, testing and validation

Some Issues in the Testing of Computer Simulation Models

The testing of simulation models has much in common with testing processes in other types of application involving software development. However, there are also important differences associated with the fact that simulation model testing involves two distinct aspects, which are known as verification and validation. Model validation is concerned with investigation of modelling errors and model limitations while verification involves checking that the simulation program is an accurate representation of the mathematical and logical structure of the underlying model. Success in model validation depends upon the availability of detailed information about all aspects of the system being modelled. It also may depend on the availability of high quality data from the system which can be used to compare its behaviour with that of the corresponding simulation model. Transparency, high standards of documentation and good management of simulation models and data sets are basic requirements in simulation model testing. Unlike most other areas of software testing, model validation often has subjective elements, with potentially important contributions from face- validation procedures in which experts give a subjective assessment of the fidelity of the model. Verification and validation processes are not simply applied once but must be used repeatedly throughout the model development process, with regressive testing principles being applied. Decisions about when a model is acceptable for the intended application inevitably involve some form of risk assessment. A case study concerned with the development and application of a simulation model of a hydro-turbine and electrical generator system is used to illustrate some of the issues arising in a typical control engineering application. Results from the case study suggest that it is important to bring together objective aspects of simulation model testing and the more subjective face- validation aspects in a coherent fashion. Suggestions are also made about the need for changes in approach in the teaching of simulation techniques to engineering students to give more emphasis to issues of model quality, testing and validation

Teaching MDE through the Formal Verification of Process Models

International audienceModel Driven Engineering (MDE) and formal methods (FM) play a key role in the development of Safety Critical Systems (SCS). They promote user oriented abstraction and formal specification using Domain Specific Modeling Languages (DSML), early Validation and formal Verification (V&V) using efficient dedicated technologies and Automatic Code and Documentation Generation. Their combined use allow to improve system qualities and reduce development costs. However, in most computer science curriculae, both domains are usually taught independently. MDE is associated to practical software engineering and FM to theoretical computer science. This contribution relates a course about MDE for SCS development that bridges the gap between these domains. It describes the content of the course and provides the lessons learned from its teaching. It focuses on early formal verification using model checking of a DSML for development process modeling. MDE technologies are illustrated both on language engineering for CASE tool development and on development process modeling. The case study also highlights the unification power of MDE as it does not target traditional executable software

Innovative teaching of IC design and manufacture using the Superchip platform

In this paper we describe how an intelligent chip architecture has allowed a large cohort of undergraduate students to be given effective practical insight into IC design by designing and manufacturing their own ICs. To achieve this, an efficient chip architecture, the “Superchip”, has been developed, which allows multiple student designs to be fabricated on a single IC, and encapsulated in a standard package without excessive cost in terms of time or resources. We demonstrate how the practical process has been tightly coupled with theoretical aspects of the degree course and how transferable skills are incorporated into the design exercise. Furthermore, the students are introduced at an early stage to the key concepts of team working, exposure to real deadlines and collaborative report writing. This paper provides details of the teaching rationale, design exercise overview, design process, chip architecture and test regime

Computer-Assisted Program Reasoning Based on a Relational Semantics of Programs

We present an approach to program reasoning which inserts between a program and its verification conditions an additional layer, the denotation of the program expressed in a declarative form. The program is first translated into its denotation from which subsequently the verification conditions are generated. However, even before (and independently of) any verification attempt, one may investigate the denotation itself to get insight into the "semantic essence" of the program, in particular to see whether the denotation indeed gives reason to believe that the program has the expected behavior. Errors in the program and in the meta-information may thus be detected and fixed prior to actually performing the formal verification. More concretely, following the relational approach to program semantics, we model the effect of a program as a binary relation on program states. A formal calculus is devised to derive from a program a logic formula that describes this relation and is subject for inspection and manipulation. We have implemented this idea in a comprehensive form in the RISC ProgramExplorer, a new program reasoning environment for educational purposes which encompasses the previously developed RISC ProofNavigator as an interactive proving assistant.Comment: In Proceedings THedu'11, arXiv:1202.453

Formal Reasoning Using an Iterative Approach with an Integrated Web IDE

This paper summarizes our experience in communicating the elements of reasoning about correctness, and the central role of formal specifications in reasoning about modular, component-based software using a language and an integrated Web IDE designed for the purpose. Our experience in using such an IDE, supported by a 'push-button' verifying compiler in a classroom setting, reveals the highly iterative process learners use to arrive at suitably specified, automatically provable code. We explain how the IDE facilitates reasoning at each step of this process by providing human readable verification conditions (VCs) and feedback from an integrated prover that clearly indicates unprovable VCs to help identify obstacles to completing proofs. The paper discusses the IDE's usage in verified software development using several examples drawn from actual classroom lectures and student assignments to illustrate principles of design-by-contract and the iterative process of creating and subsequently refining assertions, such as loop invariants in object-based code.Comment: In Proceedings F-IDE 2015, arXiv:1508.0338