On the use of testability measures for dependability assessment

Program “testability” is informally, the probability that a program will fail under test if it contains at least one fault. When a dependability assessment has to be derived from the observation of a series of failure free test executions (a common need for software subject to “ultra high reliability” requirements), measures of testability can-in theory-be used to draw inferences on program correctness. We rigorously investigate the concept of testability and its use in dependability assessment, criticizing, and improving on, previously published results. We give a general descriptive model of program execution and testing, on which the different measures of interest can be defined. We propose a more precise definition of program testability than that given by other authors, and discuss how to increase testing effectiveness without impairing program reliability in operation. We then study the mathematics of using testability to estimate, from test results: the probability of program correctness and the probability of failures. To derive the probability of program correctness, we use a Bayesian inference procedure and argue that this is more useful than deriving a classical “confidence level”. We also show that a high testability is not an unconditionally desirable property for a program. In particular, for programs complex enough that they are unlikely to be completely fault free, increasing testability may produce a program which will be less trustworthy, even after successful testin

Robust Dynamic Selection of Tested Modules in Software Testing for Maximizing Delivered Reliability

Software testing is aimed to improve the delivered reliability of the users. Delivered reliability is the reliability of using the software after it is delivered to the users. Usually the software consists of many modules. Thus, the delivered reliability is dependent on the operational profile which specifies how the users will use these modules as well as the defect number remaining in each module. Therefore, a good testing policy should take the operational profile into account and dynamically select tested modules according to the current state of the software during the testing process. This paper discusses how to dynamically select tested modules in order to maximize delivered reliability by formulating the selection problem as a dynamic programming problem. As the testing process is performed only once, risk must be considered during the testing process, which is described by the tester's utility function in this paper. Besides, since usually the tester has no accurate estimate of the operational profile, by employing robust optimization technique, we analysis the selection problem in the worst case, given the uncertainty set of operational profile. By numerical examples, we show the necessity of maximizing delivered reliability directly and using robust optimization technique when the tester has no clear idea of the operational profile. Moreover, it is shown that the risk averse behavior of the tester has a major influence on the delivered reliability.Comment: 19 pages, 4 figure

Statistical modeling of software reliability

This working paper discusses the statistical simulation part of a controlled software development experiment being conducted under the direction of the System Validation Methods Branch, Information Systems Division, NASA Langley Research Center. The experiment uses guidance and control software (GCS) aboard a fictitious planetary landing spacecraft: real-time control software operating on a transient mission. Software execution is simulated to study the statistical aspects of reliability and other failure characteristics of the software during development, testing, and random usage. Quantification of software reliability is a major goal. Various reliability concepts are discussed. Experiments are described for performing simulations and collecting appropriate simulated software performance and failure data. This data is then used to make statistical inferences about the quality of the software development and verification processes as well as inferences about the reliability of software versions and reliability growth under random testing and debugging

Software Reliability Growth Models from the Perspective of Learning Effects and Change-Point.

Increased attention towards reliability of software systems has led to the thorough analysis of the process of reliability growth for prediction and assessment of software reliability in the testing or debugging phase. With many frameworks available in terms of the underlying probability distributions like Poisson process, Non-Homogeneous Poisson Process (NHPP), Weibull, etc, many researchers have developed models using the Non-Homogeneous Poisson Process (NHPP) analytical framework. The behavior of interest, usually, is S-shaped or exponential shaped. S-shaped behavior could relate more closely to the human learning. The need to develop different models stems from the fact that nature of the underlying environment, learning effect acquisition during testing, resource allocations, application and the failure data itself vary. There is no universal model that fits everywhere to be called an Oracle. Learning effects that stem from the experiences of the testing or debugging staff have been considered for the growth of reliability. Learning varies over time and this asserts need for conduct of more research for study of learning effects.Digital copy of ThesisUniversity of Kashmi

A methodology for producing reliable software, volume 1

An investigation into the areas having an impact on producing reliable software including automated verification tools, software modeling, testing techniques, structured programming, and management techniques is presented. This final report contains the results of this investigation, analysis of each technique, and the definition of a methodology for producing reliable software

Reliability improvement and assessment of safety critical software

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Nuclear Engineering; and, (S.M.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 1998.Includes bibliographical references (leaves 95-101).In order to allow the introduction of safety-related Digital Instrumentation and Control (DI&C) systems in nuclear power plants, the software used by the systems must be demonstrated to be highly reliable. The most widely used and most powerful method for ensuring high software quality and reliability is testing. An integrated methodology is developed in this thesis for reliability assessment and improvement of safety critical software through testing. The methodology is based upon input domain-based reliability modeling and structural testing method. The purpose of the methodology is twofold: Firstly it can be used to control the testing process. The methodology provides path selection criteria and stopping criteria for the testing process with the aim to achieve maximum reliability improvement using available testing resources. Secondly, it can be used to assess and quantify the reliability of the software after the testing process. The methodology provides a systematic mechanism to quantify the reliability and estimate uncertainty of the software after testing.by Yu Sui.S.M

Automated Dynamic Firmware Analysis at Scale: A Case Study on Embedded Web Interfaces

Embedded devices are becoming more widespread, interconnected, and web-enabled than ever. However, recent studies showed that these devices are far from being secure. Moreover, many embedded systems rely on web interfaces for user interaction or administration. Unfortunately, web security is known to be difficult, and therefore the web interfaces of embedded systems represent a considerable attack surface. In this paper, we present the first fully automated framework that applies dynamic firmware analysis techniques to achieve, in a scalable manner, automated vulnerability discovery within embedded firmware images. We apply our framework to study the security of embedded web interfaces running in Commercial Off-The-Shelf (COTS) embedded devices, such as routers, DSL/cable modems, VoIP phones, IP/CCTV cameras. We introduce a methodology and implement a scalable framework for discovery of vulnerabilities in embedded web interfaces regardless of the vendor, device, or architecture. To achieve this goal, our framework performs full system emulation to achieve the execution of firmware images in a software-only environment, i.e., without involving any physical embedded devices. Then, we analyze the web interfaces within the firmware using both static and dynamic tools. We also present some interesting case-studies, and discuss the main challenges associated with the dynamic analysis of firmware images and their web interfaces and network services. The observations we make in this paper shed light on an important aspect of embedded devices which was not previously studied at a large scale. We validate our framework by testing it on 1925 firmware images from 54 different vendors. We discover important vulnerabilities in 185 firmware images, affecting nearly a quarter of vendors in our dataset. These experimental results demonstrate the effectiveness of our approach

Availability and Reliability Analysis of Computer Software Systems Considering Maintenance and Security Issues

Ph.DDOCTOR OF PHILOSOPH

Fault detection and correction modeling of software systems

Ph.DDOCTOR OF PHILOSOPH