Some conservative stopping rules for the operational testing of safety-critical software

Operational testing, which aims to generate sequences of test cases with the same statistical properties as those that would be experienced in real operational use, can be used to obtain quantitative measures of the reliability of software. In the case of safety critical software it is common to demand that all known faults are removed. This means that if there is a failure during the operational testing, the offending fault must be identified and removed. Thus an operational test for safety critical software takes the form of a specified number of test cases (or a specified period of working) that must be executed failure-free. This paper addresses the problem of specifying the numbers of test cases (or time periods) required for a test, when the previous test has terminated as a result of a failure. It has been proposed that, after the obligatory fix of the offending fault, the software should be treated as if it were completely novel, and be required to pass exactly the same test as originally specified. The reasoning here claims to be conservative, inasmuch as no credit is given for any previous failure-free operation prior to the failure that terminated the test. We show that, in fact, this is not a conservative approach in all cases, and propose instead some new Bayesian stopping rules. We show that the degree of conservatism in stopping rules depends upon the precise way in which the reliability requirement is expressed. We define a particular form of conservatism that seems desirable on intuitive grounds, and show that the stopping rules that exhibit this conservatism are also precisely the ones that seem preferable on other grounds

The use of multilegged arguments to increase confidence in safety claims for software-based systems: A study based on a BBN analysis of an idealized example

The work described here concerns the use of so-called multi-legged arguments to support dependability claims about software-based systems. The informal justification for the use of multi-legged arguments is similar to that used to support the use of multi-version software in pursuit of high reliability or safety. Just as a diverse, 1-out-of-2 system might be expected to be more reliable than each of its two component versions, so a two-legged argument might be expected to give greater confidence in the correctness of a dependability claim (e.g. a safety claim) than would either of the argument legs alone. Our intention here is to treat these argument structures formally, in particular by presenting a formal probabilistic treatment of ‘confidence’, which will be used as a measure of efficacy. This will enable claims for the efficacy of the multi-legged approach to be made quantitatively, answering questions such as ‘How much extra confidence about a system’s safety will I have if I add a verification argument leg to an argument leg based upon statistical testing?’ For this initial study, we concentrate on a simplified and idealized example of a safety system in which interest centres upon a claim about the probability of failure on demand. Our approach is to build a BBN (“Bayesian Belief Network”) model of a two-legged argument, and manipulate this analytically via parameters that define its node probability tables. The aim here is to obtain greater insight than is afforded by the more usual BBN treatment, which involves merely numerical manipulation. We show that the addition of a diverse second argument leg can, indeed, increase confidence in a dependability claim: in a reasonably plausible example the doubt in the claim is reduced to one third of the doubt present in the original single leg. However, we also show that there can be some unexpected and counter-intuitive subtleties here; for example an entirely supportive second leg can sometimes undermine an original argument, resulting overall in less confidence than came from this original argument. Our results are neutral on the issue of whether such difficulties will arise in real life - i.e. when real experts judge real systems

Some conservative stopping rules for the operational testing of saftey-critical software

Operational testing, which aims to generate sequences of test cases with the same statistical properties as those that would be experienced in real operational use, can be used to obtain quantitative measures of the reliability of software. In the case of safety critical software it is common to demand that all known faults are removed. This means that if there is a failure during the operational testing, the offending fault must be identified and removed. Thus an operational test for safety critical software takes the form of a specified number of test cases (or a specified period of working) that must be executed failure-free. This paper addresses the problem of specifiying the number of test cases (or time periods) required for a test, when the previous test has terminated as a result of a failue. It has been proposed that, after the obligatory fix of the offending fault, the software should be treated as if it were completely novel, and be required to pass exactly the same test as originally specified. The reasoning here claims to be conservative, inasmuch as no credit is given for any previous failure-free operation prior to the failure that terminated the test. We show that, in fact, this is not a conservative approach in all cases, and propose instead some new Bayesian stopping rules. We show that the degree of conservatism in stopping rules depends upon the precise way in which the reliability requirements is expressed. We show that some rules are 'completely' conservative and argue that these are also precisely the ones that should be preferred on intuitive grounds

Confidence: Its role in dependability cases for risk assessment

Society is increasingly requiring quantitative assessment of risk and associated dependability cases. Informally, a dependability case comprises some reasoning, based on assumptions and evidence, that supports a dependability claim at a particular level of confidence. In this paper we argue that a quantitative assessment of claim confidence is necessary for proper assessment of risk. We discuss the way in which confidence depends upon uncertainty about the underpinnings of the dependability case (truth of assumptions, correctness of reasoning, strength of evidence), and propose that probability is the appropriate measure of uncertainty. We discuss some of the obstacles to quantitative assessment of confidence (issues of composability of subsystem claims; of the multi-dimensional, multi-attribute nature of dependability claims; of the difficult role played by dependence between different kinds of evidence, assumptions, etc). We show that, even in simple cases, the confidence in a claim arising from a dependability case can be surprisingly low

Reliability modeling of a 1-out-of-2 system: Research with diverse Off-the-shelf SQL database servers

Fault tolerance via design diversity is often the only viable way of achieving sufficient dependability levels when using off-the-shelf components. We have reported previously on studies with bug reports of four open-source and commercial off-the-shelf database servers and later release of two of them. The results were very promising for designers of fault-tolerant solutions that wish to employ diverse servers: very few bugs caused failures in more than one server and none caused failure in more than two. In this paper we offer details of two approaches we have studied to construct reliability growth models for a 1-out-of-2 fault-tolerant server which utilize the bug reports. The models presented are of practical significance to system designers wishing to employ diversity with off-the-shelf components since often the bug reports are the only direct dependability evidence available to them

The Mumwa Crafts Association "Community Development through Craft Production"

This case study presents a detailed impact analysis of the Mumwa Crafts Association in Zambia. It considers its activities as a social purpose venture and impact in terms of poverty alleviation and sustainable development. The case study was developed as part of the ESRC funded Trickle Out Africa Project which explores social and environmental enterprises across 19 countries in sub-Saharan Africa. Cite as: Littlewood, D and Holt, D. (2013). The Mumwa Crafts Association Community Development through Craft Production". The Trickle Out Africa Project case study series: No.3. April, ISSN 2052-002

The Book Bus "Improving children's lives one book at a time"

This case study presents a detailed impact analysis of the Book Bus educational project and enterprise - with specific reference to operations in Zambia. It considers its activities as a social purpose venture and impact in terms of poverty alleviation and sustainable development. The case study was developed as part of the ESRC funded Trickle Out Africa Project which explores social and environmental enterprises across 19 countries in sub-Saharan Africa. Cite as: Holt, D. and Littlewood, D. (2013). The Book Bus "Improving children's lives one book at a time" The Trickle Out Africa Project case study series: No.2. February, ISSN 2052-002

The Khayelitsha Cookie Company "Creating opportunity one bite at a time".

This case study presents a detailed impact analysis of the Khayelitsha Cookie Company in South Africa which employs marginalised women from the local township. It considers its activities as a social purpose venture and impact in terms of poverty alleviation and sustainable development. The case study was developed as part of the ESRC funded Trickle Out Africa Project which explores social and environmental enterprises across 19 countries in sub-Saharan Africa. Cite as: Holt, D. and Littlewood, D. (2013). The Khayelitsha Cookie Company "Creating opportunity one bite at a time". The Trickle Out Africa Project case study series: No.4. May, ISSN 2052-002