The Validation of Computer-based Models in Engineering: Some Lessons from Computing Science

Questions of the quality of computer-based models and the formal processes of model testing, involving internal verification and external validation, are usually given only passing attention in engineering reports and in technical publications. However, such models frequently provide a basis for analysis methods, design calculations or real-time decision-making in complex engineering systems. This paper reviews techniques used for external validation of computer-based models and contrasts the somewhat casual approach which is usually adopted in this field with the more formal approaches to software testing and documentation recommended for large software projects. Both activities require intimate knowledge of the intended application, a systematic approach and considerable expertise and ingenuity in the design of tests. It is concluded that engineering degree courses dealing with modelling techniques and computer simulation should put more emphasis on model limitations, testing and validation

Computer Forensic Functions Testing: Media Preparation, Write Protection and Verification

The growth in the computer forensic field has created a demand for new software (or increased functionality to existing software) and a means to verify that this software is truly forensic i.e. capable of meeting the requirements of the trier of fact. In this work, we review our previous work---a function oriented testing framework for validation and verification of computer forensic tools. This framework consists of three parts: function mapping, requirements specification and reference set development. Through function mapping, we give a scientific and systemized description of the fundamentals of computer forensic discipline, i.e. what functions are needed in the computer forensic investigation process. We focus this paper on the functions of media preparation, write protection and verification. Specifically, we complete the function mapping of these functions and specify their requirements. Based on this work, future work can be conducted to develop corresponding reference sets to test any tools that possess these functions

Independent Verification and Validation of Complex User Interfaces: A Human Factors Approach

The Usability Testing and Analysis Facility (UTAF) at the NASA Johnson Space Center has identified and evaluated a potential automated software interface inspection tool capable of assessing the degree to which space-related critical and high-risk software system user interfaces meet objective human factors standards across each NASA program and project. Testing consisted of two distinct phases. Phase 1 compared analysis times and similarity of results for the automated tool and for human-computer interface (HCI) experts. In Phase 2, HCI experts critiqued the prototype tool's user interface. Based on this evaluation, it appears that a more fully developed version of the tool will be a promising complement to a human factors-oriented independent verification and validation (IV&V) process

The Validation of an Infrared Simulation System

A commonly-used term in the simulation domain is ‘validation, verification and accreditation’ (VVA). When analysing simulation predictions for the purpose of system solution development and decision-making, one key question persist: “What confidence can I have in the simulation and its results? ” Knowing the validation status of a simulation system is critical to express confidence in the simulation. A practical validation procedure must be simple and done in the regular course of work. A well-known and acknowledged validation model by Schlesinger depicts the interaction between three entities: Reality, Conceptual Model and Computer Model, and three processes: Analysis & Modelling, Programming and Verification, and Evaluation and Validation. We developed a systematic procedure where each of these six elements is evaluated, investigated and then quantified in terms of a set of criteria (or model properties). Many techniques exist to perform the validation procedure. They include: comparison with other models, face validity, extreme condition testing, historical data validation and predictive validation - to mention a few. The result is a two- dimensional matrix representing the confidence in validation of each of the criteria (model properties) along each of the verification and validation elements. Depending on the nature of the element, the quantification of each cell in this matrix is done numerically or heuristically. Most often literature on validation for simulation systems only provides guidance by means of a theoretical validation framework. This paper briefly describes the procedure used to validate software models in an infrared system simulation, and provides application examples of this process. The discussion includes practical validation techniques, quantification, visualisation, summary reports, and lessons learned during the course of a validation process. The framework presented in this paper is sufficiently general, so that the concepts could be applied to other simulation environments as well

Using KLEE to generate test cases for the Texas Instruments® Stellaris® Peripheral Driver Library

textSoftware engineers spend much of their time checking the correctness of software. Software testing is the most widely used technique for accomplishing this task. Most of the test cases used for checking software are manually created, and may not always cover all execution paths of the software. If key test cases are not executed, then the possibility of errors within the software still exists. By using tools that can automate the testing of software, software engineers can run exhaustive tests on their applications to provide verification and validation. Symbolic execution is a program analysis technique that can be utilized to achieve this. KLEE is an open-source dynamic test generation tool based on symbolic execution. In this report I present my results from evaluating KLEE on the Texas Instruments® Stellaris® Peripheral Driver Library. The Stellaris® Peripheral Driver Library consists of software drivers for controlling the peripherals on the Stellaris suite of ARM® Cortex-M based microcontrollers. In total 554 functions within the library were tested, and a total of 14763 test cases were generated. There were 32 bugs found in the software, which include assertion violations, memory errors, and arithmetic errors (division by zero, and shift errors).Electrical and Computer Engineerin

Flight control system rapid prototyping for the remotely-controlled elettra-twin-flyer airship

Nautilus S.p. A. is a small company investing in the design and development of a low-cost multipurpose multi-mission platform, known as Elettra-Twin-Flyer, which is a very innovative radio-controled airship, equipped with high precision sensors and telecommunication devices. In the prototype phase, Nautilus policy is oriented towards a massive employment of external collaborators to reduce the development costs. The crucial problem of this kind of management is the harmonious integration of all the teams involved on the project. This paper describes the integration process of the PC-104 on-board computer with the avionic devices, which are electronic systems characterized by complex communication protocols. Attention is focused on the testing, verification, validation and final translation of the embedded control software into the on-board computer, through techniques derived from the automatic code generation, such as Rapid Prototyping and Hardware-In-the-Loop. Copyright © 2006 by the American Institute of Aeronautics and Astronautics, Inc. All rights reserved

Development, analysis, and implications of open-source simulations of remotely piloted aircraft

In recent years, the use of Remotely Piloted Aircraft (RPAs) for diverse purposes has increased exponentially. As a consequence, the uncertainty created by situations turning into a threat for civilians has led to more restrictive regulations from national administrations such as Transport Canada. Their purpose is to safely integrate RPAs in the current airspace used for piloted aviation by evaluating Sense and Avoid (SAA) strategies and close encounters. The difficulty falls on having to rely on simulated environments because of the risk to the human pilot in the piloted aircraft. In the first part of this research, the technical difficulties associated with the development and study of RPA computer models are discussed. It explores the rationale behind using Open-Source Software (OSS) platforms for simulating RPAs as well as the challenges associated with interacting with OSS at graduate student level. A set of recommendations is proposed as the solution to improve the graduate student experience with OSS. In the second part, particular challenges related to the design of OSS computer models are addressed. Based on: (1) the differences and similarities between piloted and RPA flight simulators and (2) existing Verification and Validation (V&V) approaches, a validation method is presented as a solution to the subject of developing fixed-wing RPAs in OSS environments. This method is used to design two flight dynamics models with SAA applications. The first computer model is presented in tutorial format as a case study for the validation procedure whereas the second computer model is specific for testing SAA strategies. In the last part, one of the designed RPAs is integrated into a computer environment with a representative general aircraft. From the simulated encounters, a diving avoidance manoeuvre on the RPA is developed. This performance is observed to analyze the consequences to the airspace. The implications of this research are seen from three perspectives: (1) the OSS challenges in graduate school are wide-spread across disciplines, (2) the proposed validation procedure is adaptable to fit any computer model and simulation scenario, and (3) the simulated OSS framework with an RPA computer model has served for testing preliminary SAA methods with close encounters with manned aircraft