8. Issues in Intelligent Computer-Assisted Instruction: Eval uation and Measurement

In this chapter we plan to explore two issues in the field of intelligent computer assisted instruction (ICAI) that we feel offer opportunities to advance the state of the art. These issues are evaluation of ICAI systems and the use of the underlying technology in ICAI systems to develop tests. For each issue we will provide a theoretical context, discuss key constructs, provide a brief window to the appropriate literature, suggest methodological solutions and conclude with a concrete example of the feasibility of the solution from our own research. INTELLIGENT COMPUTER-ASSISTED INSTRUCTION (ICAI) ICAI is the application of artificial intelligence to computer-assisted instruction. Artificial intelligence, a branch of computer science, is making computers smart in order to (a) make them more useful and (b) understand intelligence (Winston, 1977). Topic areas in artificial intelligence have included natural language processing (Schank, 1980), vision (Winston, 1975), knowledge representation (Woods, 1983), spoken language (Lea, 1980), planning (Hayes-Roth, 1980), and expert systems (Buchanan, 1981). The field of Artificial Intelligence (AI) has matured in both hardware and software. The most commonly used language in the field is LISP (List Processing). A major development in the hardware area is that personal LISP machines are now available at a relatively low cost (20-50K) with the power of prior mainframes. In the software area two advances stand out: (a) programming support environments such as LOOPS (Bobrow & Stefik, 1983) and (b) expert system tools. These latter tools are now running on powerful micros. The application of expert systems technology to a host of real-world problems has demonstrated the utility of artificial intelligence techniques in a very dramatic style. Expert system technology is the branch of artificial intelligence at this point most relevant to ICAI. Expert Systems Knowledge-based systems or expert systems are a collection of problem-solving computer programs containing both factual and experiential knowledge and data in a particular domain. When the knowledge embodied in the program is a result of a human expert elicitation, these systems are called expert systems. A typical expert system consists of a knowledge base, a reasoning mechanism popularly called an inference engine and a friendly user interface. The knowledge base consists of facts, concepts, and numerical data (declarative knowledge), procedures based on experience or rules of thumb (heuristics), and causal or conditional relationships (procedural knowledge). The inference engine searches or reasons with or about the knowledge base to arrive at intermediate conclusions or final results during the course of problem solving. It effectively decides when and what knowledge should be applied, applies the knowledge and determines when an acceptable solution has been found. The inference engine employs several problem-solving strategies in arriving at conclusions. Two of the popular schemes involve starting with a good description or desired solution and working backwards to the known facts or current situation (backward chaining), and starting with the current situation or known facts and working toward a goal or desired solution (forward chaining). The user interface may give the user choices (typically menu-driven) or allow the user to participate in the control of the process (mixed initiative). The interface allows the user: to describe a problem, input knowledge or data, browse through the knowledge base, pose question, review the reasoning process of the system, intervene as necessary, and control overall system operation. Successful expert systems have been developed in fields as diverse as mineral exploration (Duda & Gaschnig, 1981) and medical diagnosis (Clancy, 1981). ICAI Systems ICAI systems use approaches artificial intelligence and cognitive science to teach a range of subject matters. Representative types of subjects include: (a) collection of facts, for example, South American geography in SCHOLAR (Carbonell & Collins, 1973); (b) complete system models, for example, a ship propulsion system in STEAMER (Stevens & Steinberg, 1981) and a power supply in SOPHIE (Brown, Burton, & de Kleer, 1982); (c) completely described procedural rules, for example, strategy learning, WEST (Brown, Burton, & de Kleer, 1982), or arithmetic in BUGGY (Brown & Burton, 1978); (d) partly described procedural rules, for example, computer programming in PROUST (Johnson & Soloway, 1983); LISP Tutor (Anderson, Boyle, & Reiser, 1985); rules in ALGEBRA (McArthur, Stasz, & Hotta, 1987); diagnosis of infectious diseases in GUIDON (Clancey, 1979); and an imperfectly understood complex domain, causes of rainfall in WHY (Stevens, Collins, & Goldin, 1978). Excellent reviews by Barr and Feigenbaum (1982) and Wenger (1987) document many of these ICAI systems. Representative research in ICAI is described by O\u27Neil, Anderson, and Freeman (1986) and Wenger (1987). Although suggestive evidence has been provided by Anderson et al. (1985), few of these ICAI projects have been evaluated in any rigorous fashion. In a sense they have all been toy systems for research and demonstration. Yet, they have raised a good deal of excitement and enthusiasm about their likelihood of being effective instructional environments. With respect to cognitive science, progress has been made in the following areas: identification and analysis of misconceptions or bugs (Clement, Lockhead, & Soloway, 1980), the use of learning strategies (O\u27Neil & Spielberger, 1979; Weinstein & Mayer, 1986), expert versus novice distinction (Chi, Glaser, & Rees, 1982), the role of mental models in learning (Kieras & Bovair, 1983), and the role of self-explanations in problem solving (Chi, Bassok, Lewis, Reimann, & Glaser, 1987). The key components of an ICAI system consist of a knowledge base: that is, (a) what the student is to learn; (b) a student model, either where the student is now with respect to subject matter or how student characteristics interact with subject matters, and (c) a tutor, that is, instructional techniques for teaching the declarative or procedural knowledge. These components are described in more detail by Fletcher (1985)

A formal definition for the expressive power of knowledge representation languages

The notions "expressive power" or "expressiveness" of knowledge representation languages (KR-languages) can be found in most papers on knowledge representation; but these terms are usually just used in an intuitive sense. The papers contain only informal descriptions of what is meant by expressiveness. There are several reasons which speak in favour of a formal definition of expressiveness: For example, if we want to show that certain expressions in one language cannot be expressed in another language, we need a strict formalism which can be used in mathematical proofs. Though we shall only consider KL-ONE-based KR-language in our motivation and in the examples, the definition of expressive power which will be given in this paper can be used for all KR-languages with model-theoretic semantics. This definition will shed a new light on the tradeoff between expressiveness of a representation language and its computational tractability. There are KR-languages with identical expressive power, but different complexity results for reasoning. Sometimes, the tradeoff lies between convenience and computational tractability. The paper contains several examples which demonstrate how the definition of expressive power can be used in positive proofs -- that is, proofs where it is shown that one language can be expressed by another language -- as well as for negative proofs -- which show that a given language cannot be expressed by the other language

A STUDY IN THE INFORMATION CONTENT, CONSISTENCY, AND EXPRESSIVE POWER OF FUNCTION STRUCTURES IN MECHANICAL DESIGN

In engineering design research, function structures are used to represent the intended functionality of technical artifacts. Function structures are graph-based representations where the nodes are functions, or actions, and the edges are flows, or objects of those actions. For the consistent description of artifact functionality, multiple controlled vocabularies have been developed in previous research. The Functional Basis is one such vocabulary that provides for a set of verbs and a set of nouns, organized in the three-level hierarchy. This vocabulary is extensively studied in design research. Two major application of this vocabulary are the Design Repository, which is a web-base archive of design information of consumer electro-mechanical products obtained through reverse engineering, and the functional decomposition grammar rules that synthesizes sub-functions or elementary actions of a product from the overall function or goal of the product. However, despite the Functional Basis\u27 popularity, the usefulness of its hierarchical structure has not been specifically tested. Additionally, although this vocabulary provides the verbs and nouns, no explicit guideline for using those terms in function structures has been proposed. Consequently, multiple representational inconsistencies can be found in the function structures within the Design Repository. The two research goals in this thesis are: (1) to investigate if the hierarchy in the Functional Basis is useful for constructing function structures and (2) to explore means to increase the consistency and expressive power of the Functional Basis vocabulary. To address the first goal, an information metric for function structures and function vocabularies is developed based on the principles of Information Theory. This metric is applied to three function structures from the Design Repository to demonstrate that the secondary level of the Functional Basis is the most informative of the three. This finding is validated by an external empirical study, which shows that the secondary level is used most frequently in the Design Repository, finally indicating that the hierarchy is not useful for constructing function structures. To address the second research goal, a new representation of functions, including rules the topological connections in a function structure, is presented. It is demonstrated through experiments that the new representation is more expressive than the text-based descriptions of functions used in the Functional Basis, as it formally describes which flows can be connected to which functions. It is also shown that the new representation reduces the uncertainty involved in the individual function structures

A formal definition for the expressive power of knowledge representation languages

The notions "expressive power" or "expressiveness" of knowledge representation languages (KR-languages) can be found in most papers on knowledge representation; but these terms are usually just used in an intuitive sense. The papers contain only informal descriptions of what is meant by expressiveness. There are several reasons which speak in favour of a formal definition of expressiveness: For example, if we want to show that certain expressions in one language cannot be expressed in another language, we need a strict formalism which can be used in mathematical proofs. Though we shall only consider KL-ONE-based KR-language in our motivation and in the examples, the definition of expressive power which will be given in this paper can be used for all KR-languages with model-theoretic semantics. This definition will shed a new light on the tradeoff between expressiveness of a representation language and its computational tractability. There are KR-languages with identical expressive power, but different complexity results for reasoning. Sometimes, the tradeoff lies between convenience and computational tractability. The paper contains several examples which demonstrate how the definition of expressive power can be used in positive proofs -- that is, proofs where it is shown that one language can be expressed by another language -- as well as for negative proofs -- which show that a given language cannot be expressed by the other language

Spatial contexts : an interactive environment for personal design

Thesis (M.S.V.S.)--Massachusetts Institute of Technology, Dept. of Architecture, 1986.MICROFICHE COPY AVAILABLE IN ARCHIVES AND ROTCHBibliography: leaves 133-140.by Jonathan Scott Linowes.M.S.V.S