Design for validation: An approach to systems validation

Every complex system built is validated in some manner. Computer validation begins with review of the system design. As systems became too complicated for one person to review, validation began to rely on the application of adhoc methods by many individuals. As the cost of the changes mounted and the expense of failure increased, more organized procedures became essential. Attempts at devising and carrying out those procedures showed that validation is indeed a difficult technical problem. The successful transformation of the validation process into a systematic series of formally sound, integrated steps is necessary if the liability inherent in the future digita-system-based avionic and space systems is to be minimized. A suggested framework and timetable for the transformtion are presented. Basic working definitions of two pivotal ideas (validation and system life-cyle) are provided and show how the two concepts interact. Many examples are given of past and present validation activities by NASA and others. A conceptual framework is presented for the validation process. Finally, important areas are listed for ongoing development of the validation process at NASA Langley Research Center

Towards Accurate Estimation of Error Sensitivity in Computer Systems

Fault injection is an increasingly important method for assessing, measuringand observing the system-level impact of hardware and software faults in computer systems. This thesis presents the results of a series of experimental studies in which fault injection was used to investigate the impact of bit-flip errors on program execution. The studies were motivated by the fact that transient hardware faults in microprocessors can cause bit-flip errors that can propagate to the microprocessors instruction set architecture registers and main memory. As the rate of such hardware faults is expected to increase with technology scaling, there is a need to better understand how these errors (known as ‘soft errors’) influence program execution, especially in safety-critical systems.Using ISA-level fault injection, we investigate how five aspects, or factors, influence the error sensitivity of a program. We define error sensitivity as the conditional probability that a bit-flip error in live data in an ISA-register or main-memory word will cause a program to produce silent data corruption (SDC; i.e., an erroneous result). We also consider the estimation of a measure called SDC count, which represents the number of ISA-level bit flips that cause an SDC.The five factors addressed are (a) the inputs processed by a program, (b) the level of compiler optimization, (c) the implementation of the program in the source code, (d) the fault model (single bit flips vs double bit flips) and (e)the fault-injection technique (inject-on-write vs inject-on-read). Our results show that these factors affect the error sensitivity in many ways; some factors strongly impact the error sensitivity or SDC count whereas others show a weaker impact. For example, our experiments show that single bit flips tend to cause SDCs more than double bit flips; compiler optimization positively impacts the SDC count but not necessarily the error sensitivity; the error sensitivity varies between 20% and 50% among the programs we tested; and variations in input affect the error sensitivity significantly for most of the tested programs

A survey on online monitoring approaches of computer-based systems

This report surveys forms of online data collection that are in current use (as well as being the subject of research to adapt them to changing technology and demands), and can be used as inputs to assessment of dependability and resilience, although they are not primarily meant for this use

Stochastic modeling for performance evaluation of database replication protocols

Performance is often the most important non-functional property for database systems and associated replication solutions. This is true at least in in-dustrial contexts. Evaluating performance using real systems, however, is com-putationally demanding and costly. In many cases, choosing between several competing replication protocols poses a difficulty in ranking these protocols meaningfully: the ranking is determined not so much by the quality of the com-peting protocols but, instead, by the quality of the available implementations. Addressing this difficulty requires a level of abstraction in which the impact on the comparison of the implementations is reduced, or entirely eliminated. We propose a stochastic model for performance evaluation of database replication protocols, paying particular attention to: i) empirical validation of a number of assumptions used in the stochastic model, and ii) empirical validation of model accuracy for a chosen replication protocol. For the empirical validations we used the TPC-C benchmark. Our implementation of the model is based on Stochastic Activity Networks (SAN), extended by bespoke code. The model may reduce the cost of performance evaluation in comparison with empirical measurements, while keeping the accuracy of the assessment to an acceptable level

Aspect-oriented fault tolerance for real-time embedded systems

Real-time embedded systems for safety-critical applications have to introduce fault tolerance mechanisms in order to cope with hardware and software errors. Fault tolerance is usually applied by means of redundancy and diversity. Redundant hardware implies the establishment of a distributed system executing a set of fault tolerance strategies by software, and may also employ some form of diversity, by using different variants or versions for the same processing. This paper describes our approach to introduce fault tolerance in distributed embedded systems applications, using aspect-oriented programming (AOP). A real-time operating system sup-porting middleware thread communication was integrated to a fault tolerant framework. The introduction of fault tolerance in the system is performed by AOP at the application thread level. The advantages of this approach include higher modularization, less efforts for legacy systems evolution and better configurability for testing and product line development. This work has been tested and evaluated successfully in several fault tolerant configurations and presented no significant performance or memory footprint costs.Fundação para a Ciência e a Tecnologia (FCT

The development of a model for predicting passenger acceptance of short-haul air transportation systems

Meaningful criteria and methodology for assessing, particularly in the area of ride quality, the potential acceptability to the traveling public of present and future transportation systems were investigated. Ride quality was found to be one of the important variables affecting the decision of users of air transportation, and to be influenced by several environmental factors, especially motion, noise, pressure, temperature, and seating. Models were developed to quantify the relationship of subjective comfort to all of these parameters and then were exercised for a variety of situations. Passenger satisfaction was found to be strongly related to ride quality and was so modeled. A computer program was developed to assess the comfort and satisfaction levels of passengers on aircraft subjected to arbitrary flight profiles over arbitrary terrain. A model was deduced of the manner in which passengers integrate isolated segments of a flight to obtain an overall trip comfort rating. A method was established for assessing the influence of other links (e.g., access, terminal conditions) in the overall passenger trip

A framework for Operational Security Metrics Development for industrial control environment

Security metrics are very crucial towards providing insights when measuring security states and susceptibilities in industrial operational environments. Obtaining practical security metrics depend on effective security metrics development approaches. To be effective, a security metrics development framework should be scope-definitive, objective-oriented, reliable, simple, adaptable, and repeatable (SORSAR). A framework for Operational Security Metrics Development (OSMD) for industry control environments is presented, which combines concepts and characteristics from existing approaches. It also adds the new characteristic of adaptability. The OSMD framework is broken down into three phases of: target definition, objective definition, and metrics synthesis. A case study scenario is used to demonstrate an instance of how to implement and apply the proposed framework to demonstrate its usability and workability. Expert elicitation has also be used to consolidate the validity of the proposed framework. Both validation approaches have helped to show that the proposed framework can help create effective and efficient ICS-centric security metrics taxonomy that can be used to evaluate capabilities or vulnerabilities. The understanding from this can help enhance security assurance within industrial operational environments