On System Scalability

A significant number of systems fail in initial use, or even during integration, because factors that have a negligible effect when systems are lightly used have a harmful effect as the level of use increases. This scalability problem (i.e., the inability of a system to accommodate an increased workload) is not new. However, the increasing size (more lines of code, greater number of users, widened scope of demands, and the like) of U.S. Department of Defense systems makes the problem more critical today than in the past. This technical note presents an analysis of what is meant by scalability and a description of factors to be considered when assessing the potential for system scalability. The factors to be considered are captured in a scalability audit, a process intended to expose issues that, if overlooked, can lead to scalability problems

Eliminative Argumentation: A Basis for Arguing Confidence in System Properties

<p>Assurance cases provide a structured method of explaining why a system has some desired property, for example, that the system is safe. But there is no agreed approach for explaining what degree of confidence one should have in the conclusions of such a case. This report defines a new concept, eliminative argumentation, that provides a philosophically grounded basis for assessing how much confidence one should have in an assurance case argument. This report will be of interest mainly to those familiar with assurance case concepts and who want to know why one argument rather than another provides more confidence in a claim. The report is also potentially of value to those interested more generally in argumentation theory.</p

Towards an Assurance Case Practice for Medical Devices

http://www.sei.cmu.edu/library/abstracts/reports/09tn018.cf

Dependability Cases

Many large software systems display fragility or a lack of dependability caused by inattention to details at various stages of development (e.g., missing data, undocumented assumptions, lack of testing), resulting in a failure to catch errors. This technical note explains how to create a dependability case for a system that helps identify and keep track of such details. A dependability case is defined here as a structured argument providing evidence that a system meets its specified dependability requirements. The technical note describes how to structure the argument and present evidence to support it. A sample problem is presented, as well as issues raised by that problem and future goals

Toward a Theory of Assurance Case Confidence

<p>Assurance cases provide an argument and evidence explaining why a claim about some system property holds. This report outlines a framework for justifying confidence in the truth of such an assurance case claim. The framework is based on the notion of eliminative induction-the principle first put forward by Francis Bacon that confidence in the truth of a hypothesis or claim increases as reasons for doubting its truth are identified and eliminated. Possible reasons for doubting the truth of a claim arise from analyzing an assurance case using defeasible reasoning concepts. Finally, the notion of Baconian probability provides a measure of confidence based on how many defeaters have been identified and eliminated.</p

Ada Adoption Handbook: A Program Manager’s Guide

The ADA Adoption Handbook provides program managers with information about how best to tap ADA's strengths and manage the transition to fully using this software technology. Although the issues are complex, they are not all unique to ADA. Indeed, many of the issues addressed in this handbook must be addressed when developing any software-intensive system in any programming language. The handbook addresses the advantages and risks in adopting ADA. Significant emphasis has been placed on providing information and suggesting methods that will help program and project managers succeed in using ADA across a broad range of application domains. The handbook focuses on the following topics: ADA's goals and benefits; program management issues; implications for education and training; software tools with emphasis on compiler validation and quality issues; the state of ADA technology as it related to system design and implementation; and the pending update of the ADA language standard (ADA 9X)

Evaluating and Mitigating Software Supply Chain Security Risks

The Department of Defense (DoD) is concerned that security vulnerabilities could be inserted into software that has been developed outside of the DoD's supervision or control. This report presents an initial analysis of how to evaluate and mitigate the risk that such unauthorized insertions have been made. The analysis is structured in terms of actions that should be taken in each phase of the DoD acquisition life cycle

Survivability Assurance for System of Systems

Complexity and change pervade today's organizations. Organizational and technology components that must work together may be created, managed, and maintained by different entities. Net-centric operations and service-oriented architectures will push this trend further, increasing the layers of people, processes, and systems. Existing analysis mechanisms do not provide a way to (1) focus on challenges arising from integrating multiple systems, (2) consider architecture tradeoffs carrying impacts beyond a single system, and (3) consider the linkage of technology to critical organizational functions. In response, a team at the Software Engineering Institute (SEI) built an analysis framework to evaluate the quality of the linkage among roles, dependencies, constraints, and risks for critical technology capabilities in the face of change. The Survivability Analysis Framework (SAF), a structured view of people, process, and technology, was developed to help organizations analyze and understand stresses and gaps to survivability for operational and proposed business processes. The SAF is designed to * identify potential problems with existing or near-term interoperations among components within today's network environments * highlight the impact on survivability as constrained interoperation moves to more dynamic connectivity * increase assurance that mission threads can survive in the presence of stress and possible failur

Improving Quality Using Architecture Fault Analysis with Confidence Arguments

<p>This case study shows how an analytical architecture fault-modeling approach can be combined with confidence arguments to diagnose a time-sensitive design error in a control system and to provide evidence that proposed changes to the system address the problem. The analytical approach, based on the SAE Architecture Analysis and Design Language for its well-defined timing and fault behavior semantics, demonstrates that such hard-to-test errors can be discovered and corrected early in the lifecycle, thereby reducing rework cost. The case study shows that by combining the analytical approach with confidence maps, we can present a structured argument that system requirements have been met and problems in the design have been addressed adequately—increasing our confidence in the system quality. The case study analyzes an aircraft engine control system that manages fuel flow with a stepper motor. The original design was developed and verified in a commercial model-based development environment without discovering the potential for missed step commanding. During system tests, actual fuel flow did not correspond to the desired fuel flow under certain circumstances. The problem was traced to missed execution of commanded steps due to variation in execution time.</p

Reliability Improvement and Validation Framework

<p>Software-reliant systems such as rotorcraft and other aircraft have experienced exponential growth in software size and complexity. The current software engineering practice of "build then test" has made them unaffordable to build and qualify. This report discusses the challenges of qualifying such systems, presenting the findings of several government and industry studies. It identifies several root cause areas and proposes a framework for reliability validation and improvement that integrates several recommended technology solutions: validation of formalized requirements; an architecture-centric, model-based engineering approach that uncovers system-level problems early through analysis; use of static analysis for validating system behavior and other system properties; and managed confidence in qualification through system assurance. This framework also provides the basis for a set of metrics for cost-effective reliability improvement that overcome the challenges of existing software complexity, reliability, and cost metrics.</p