Use of metaknowledge in the verification of knowledge-based systems

Knowledge-based systems are modeled as deductive systems. The model indicates that the two primary areas of concern in verification are demonstrating consistency and completeness. A system is inconsistent if it asserts something that is not true of the modeled domain. A system is incomplete if it lacks deductive capability. Two forms of consistency are discussed along with appropriate verification methods. Three forms of incompleteness are discussed. The use of metaknowledge, knowledge about knowledge, is explored in connection to each form of incompleteness

Verification issues for rule-based expert systems

Verification and validation of expert systems is very important for the future success of this technology. Software will never be used in non-trivial applications unless the program developers can assure both users and managers that the software is reliable and generally free from error. Therefore, verification and validation of expert systems must be done. The primary hindrance to effective verification and validation is the use of methodologies which do not produce testable requirements. An extension of the flight technique panels used in previous NASA programs should provide both documented requirements and very high levels of verification for expert systems

Approaches to the verification of rule-based expert systems

Expert systems are a highly useful spinoff of artificial intelligence research. One major stumbling block to extended use of expert systems is the lack of well-defined verification and validation (V and V) methodologies. Since expert systems are computer programs, the definitions of verification and validation from conventional software are applicable. The primary difficulty with expert systems is the use of development methodologies which do not support effective V and V. If proper techniques are used to document requirements, V and V of rule-based expert systems is possible, and may be easier than with conventional code. For NASA applications, the flight technique panels used in previous programs should provide an excellent way to verify the rules used in expert systems. There are, however, some inherent differences in expert systems that will affect V and V considerations

Building validation tools for knowledge-based systems

The Expert Systems Validation Associate (EVA), a validation system under development at the Lockheed Artificial Intelligence Center for more than a year, provides a wide range of validation tools to check the correctness, consistency and completeness of a knowledge-based system. A declarative meta-language (higher-order language), is used to create a generic version of EVA to validate applications written in arbitrary expert system shells. The architecture and functionality of EVA are presented. The functionality includes Structure Check, Logic Check, Extended Structure Check (using semantic information), Extended Logic Check, Semantic Check, Omission Check, Rule Refinement, Control Check, Test Case Generation, Error Localization, and Behavior Verification

Methodology for testing and validating knowledge bases

A test and validation toolset developed for artificial intelligence programs is described. The basic premises of this method are: (1) knowledge bases have a strongly declarative character and represent mostly structural information about different domains, (2) the conditions for integrity, consistency, and correctness can be transformed into structural properties of knowledge bases, and (3) structural information and structural properties can be uniformly represented by graphs and checked by graph algorithms. The interactive test and validation environment have been implemented on a SUN workstation

A tool-supported approach to inter-tabular verification.

The use of decision tables to verify KBS has been advocated several times in the V&V literature. However, one of the main drawbacks of those system is that they fail to detect anomalies which occur over rule chains. In a decision table based context this means that anomalies which occur due to interactions between tables are neglected. These anomalies are called inter-tabular anomalies. In this paper we investigate an approach that deals with inter-tabular anomalies. One of the prerequisites for the approach was that it could be used by the knowledge engineer during the development of the KBS. This requires that the anomaly check can be performed on-line. As a result, the approach partly uses heuristics where exhaustive checks would be too inefficient. All detection facilities that will be described, have been implemented in a table-based development tool called PROLOGA. The use of this tool will be briefly illustrated. In addition, some experiences in verifying large knowledge bases are discussed.

Validation of highly reliable, real-time knowledge-based systems

Knowledge-based systems have the potential to greatly increase the capabilities of future aircraft and spacecraft and to significantly reduce support manpower needed for the space station and other space missions. However, a credible validation methodology must be developed before knowledge-based systems can be used for life- or mission-critical applications. Experience with conventional software has shown that the use of good software engineering techniques and static analysis tools can greatly reduce the time needed for testing and simulation of a system. Since exhaustive testing is infeasible, reliability must be built into the software during the design and implementation phases. Unfortunately, many of the software engineering techniques and tools used for conventional software are of little use in the development of knowledge-based systems. Therefore, research at Langley is focused on developing a set of guidelines, methods, and prototype validation tools for building highly reliable, knowledge-based systems. The use of a comprehensive methodology for building highly reliable, knowledge-based systems should significantly decrease the time needed for testing and simulation. A proven record of delivering reliable systems at the beginning of the highly visible testing and simulation phases is crucial to the acceptance of knowledge-based systems in critical applications

Evidence flow graph methods for validation and verification of expert systems

The results of an investigation into the use of evidence flow graph techniques for performing validation and verification of expert systems are given. A translator to convert horn-clause rule bases into evidence flow graphs, a simulation program, and methods of analysis were developed. These tools were then applied to a simple rule base which contained errors. It was found that the method was capable of identifying a variety of problems, for example that the order of presentation of input data or small changes in critical parameters could affect the output from a set of rules

Considerations in development of expert systems for real-time space applications

Over the years, demand on space systems has increased tremendously and this trend will continue for the near future. Enhanced capabilities of space systems, however, can only be met with increased complexity and sophistication of onboard and ground systems. Artificial Intelligence and expert system techniques have great potential in space applications. Expert systems could facilitate autonomous decision making, improve in-orbit fault diagnosis and repair, enhance performance and reduce reliance on ground support. However, real-time expert systems, unlike conventional off-line consultative systems, have to satisfy certain special stringent requirements before they could be used for onboard space applications. Challenging and interesting new environments are faced while developing expert system space applications. This paper discusses the special characteristics, requirements and typical life cycle issues for onboard expert systems. Further, it also describes considerations in design, development, and implementation which are particularly important to real-time expert systems for space applications

Validation and verification of expert systems

Validation and verification (V&V) are procedures used to evaluate system structure or behavior with respect to a set of requirements. Although expert systems are often developed as a series of prototypes without requirements, it is not possible to perform V&V on any system for which requirements have not been prepared. In addition, there are special problems associated with the evaluation of expert systems that do not arise in the evaluation of conventional systems, such as verification of the completeness and accuracy of the knowledge base. The criticality of most NASA missions make it important to be able to certify the performance of the expert systems used to support these mission. Recommendations for the most appropriate method for integrating V&V into the Expert System Development Methodology (ESDM) and suggestions for the most suitable approaches for each stage of ESDM development are presented