8. Issues in Intelligent Computer-Assisted Instruction: Eval uation and Measurement

In this chapter we plan to explore two issues in the field of intelligent computer assisted instruction (ICAI) that we feel offer opportunities to advance the state of the art. These issues are evaluation of ICAI systems and the use of the underlying technology in ICAI systems to develop tests. For each issue we will provide a theoretical context, discuss key constructs, provide a brief window to the appropriate literature, suggest methodological solutions and conclude with a concrete example of the feasibility of the solution from our own research. INTELLIGENT COMPUTER-ASSISTED INSTRUCTION (ICAI) ICAI is the application of artificial intelligence to computer-assisted instruction. Artificial intelligence, a branch of computer science, is making computers smart in order to (a) make them more useful and (b) understand intelligence (Winston, 1977). Topic areas in artificial intelligence have included natural language processing (Schank, 1980), vision (Winston, 1975), knowledge representation (Woods, 1983), spoken language (Lea, 1980), planning (Hayes-Roth, 1980), and expert systems (Buchanan, 1981). The field of Artificial Intelligence (AI) has matured in both hardware and software. The most commonly used language in the field is LISP (List Processing). A major development in the hardware area is that personal LISP machines are now available at a relatively low cost (20-50K) with the power of prior mainframes. In the software area two advances stand out: (a) programming support environments such as LOOPS (Bobrow & Stefik, 1983) and (b) expert system tools. These latter tools are now running on powerful micros. The application of expert systems technology to a host of real-world problems has demonstrated the utility of artificial intelligence techniques in a very dramatic style. Expert system technology is the branch of artificial intelligence at this point most relevant to ICAI. Expert Systems Knowledge-based systems or expert systems are a collection of problem-solving computer programs containing both factual and experiential knowledge and data in a particular domain. When the knowledge embodied in the program is a result of a human expert elicitation, these systems are called expert systems. A typical expert system consists of a knowledge base, a reasoning mechanism popularly called an inference engine and a friendly user interface. The knowledge base consists of facts, concepts, and numerical data (declarative knowledge), procedures based on experience or rules of thumb (heuristics), and causal or conditional relationships (procedural knowledge). The inference engine searches or reasons with or about the knowledge base to arrive at intermediate conclusions or final results during the course of problem solving. It effectively decides when and what knowledge should be applied, applies the knowledge and determines when an acceptable solution has been found. The inference engine employs several problem-solving strategies in arriving at conclusions. Two of the popular schemes involve starting with a good description or desired solution and working backwards to the known facts or current situation (backward chaining), and starting with the current situation or known facts and working toward a goal or desired solution (forward chaining). The user interface may give the user choices (typically menu-driven) or allow the user to participate in the control of the process (mixed initiative). The interface allows the user: to describe a problem, input knowledge or data, browse through the knowledge base, pose question, review the reasoning process of the system, intervene as necessary, and control overall system operation. Successful expert systems have been developed in fields as diverse as mineral exploration (Duda & Gaschnig, 1981) and medical diagnosis (Clancy, 1981). ICAI Systems ICAI systems use approaches artificial intelligence and cognitive science to teach a range of subject matters. Representative types of subjects include: (a) collection of facts, for example, South American geography in SCHOLAR (Carbonell & Collins, 1973); (b) complete system models, for example, a ship propulsion system in STEAMER (Stevens & Steinberg, 1981) and a power supply in SOPHIE (Brown, Burton, & de Kleer, 1982); (c) completely described procedural rules, for example, strategy learning, WEST (Brown, Burton, & de Kleer, 1982), or arithmetic in BUGGY (Brown & Burton, 1978); (d) partly described procedural rules, for example, computer programming in PROUST (Johnson & Soloway, 1983); LISP Tutor (Anderson, Boyle, & Reiser, 1985); rules in ALGEBRA (McArthur, Stasz, & Hotta, 1987); diagnosis of infectious diseases in GUIDON (Clancey, 1979); and an imperfectly understood complex domain, causes of rainfall in WHY (Stevens, Collins, & Goldin, 1978). Excellent reviews by Barr and Feigenbaum (1982) and Wenger (1987) document many of these ICAI systems. Representative research in ICAI is described by O\u27Neil, Anderson, and Freeman (1986) and Wenger (1987). Although suggestive evidence has been provided by Anderson et al. (1985), few of these ICAI projects have been evaluated in any rigorous fashion. In a sense they have all been toy systems for research and demonstration. Yet, they have raised a good deal of excitement and enthusiasm about their likelihood of being effective instructional environments. With respect to cognitive science, progress has been made in the following areas: identification and analysis of misconceptions or bugs (Clement, Lockhead, & Soloway, 1980), the use of learning strategies (O\u27Neil & Spielberger, 1979; Weinstein & Mayer, 1986), expert versus novice distinction (Chi, Glaser, & Rees, 1982), the role of mental models in learning (Kieras & Bovair, 1983), and the role of self-explanations in problem solving (Chi, Bassok, Lewis, Reimann, & Glaser, 1987). The key components of an ICAI system consist of a knowledge base: that is, (a) what the student is to learn; (b) a student model, either where the student is now with respect to subject matter or how student characteristics interact with subject matters, and (c) a tutor, that is, instructional techniques for teaching the declarative or procedural knowledge. These components are described in more detail by Fletcher (1985)

An Object-oriented natural language expert system

This paper highlights the design and implementation of a prototype object-oriented Intelligent Tutoring System (ITS). There were two main goals for the prototype. One was to create a small tutoring system which fit into the niche of current commercially available systems, while exhibiting some of the more intelligent features found in larger custom systems. The other was to experiment with an object-oriented model, design approach, and implementation for such a system. A masking technique for pattern matching is one of the key paradigms driving the intelligent diagnosis. The prototype provides a means for practicing sentence construction, by checking a student\u27s translation responses, and offering hints, if desired. It was implemented in C++

Ask-Elle:an Adaptable Programming Tutor for Haskell Giving Automated Feedback

Ask-Elle is a tutor for learning the higher-order, strongly-typed functional programming language Haskell. It supports the stepwise development of Haskell programs by verifying the correctness of incomplete programs, and by providing hints. Programming exercises are added to Ask-Elle by providing a task description for the exercise, one or more model solutions, and properties that a solution should satisfy. The properties and model solutions can be annotated with feedback messages, and the amount of flexibility that is allowed in student solutions can be adjusted. The main contribution of our work is the design of a tutor that combines (1) the incremental development of different solutions in various forms to a programming exercise with (2) automated feedback and (3) teacher-specified programming exercises, solutions, and properties. The main functionality is obtained by means of strategy-based model tracing and property-based testing. We have tested the feasibility of our approach in several experiments, in which we analyse both intermediate and final student solutions to programming exercises, amongst others

An application of formal semantics to student modelling : an investigation in the domain of teaching Prolog

This thesis reports on research undertaken in an exploration of the use of formal semantics for student modelling in intelligent tutoring systems. The domain chosen was that of tutoring programming languages and within that domain Prolog was selected to be the target language for this exploration. The problem considered is one of how to analyse students' errors at a level which allows diagnosis to be more flexible and meaningful than is possible with the 'mal-rules' and 'bugcatalogue' approach of existing systems. The ideas put forward by Robin Milner [1980] in his Calculus of Communicating Systems (CCS) form the basis of the formalism which is proposed as a solution to this problem. Based on the findings of an empirical investigation, novices' misconceptions of control flow in Prolog was defined as a suitable area in which to explore the application of this solution. A selection of Prolog programs used in that investigation was formally described in terms of CCS. These formal descriptions were used by a production rule system to generate a number of the incomplete or faulty models of Prolog execution which were identified in the first empirical study. In a second empirical study, a machine-analysis tool, designed to be part of a diagnostic tutoring module, used these models to diagnose students' misconceptions of Prolog control flow. This initial application of CCS to student modelling showed that the models of Prolog execution generated by the system could be used successfully to detect students' misunderstandings. Results from the research reported here indicate that the use of formal semantics to model programming languages has a useful contribution to make to the task of student modelling

A methodology for evaluating intelligent tutoring systems

DissertationThis dissertation proposes a generic methodology for evaluating intelligent tutoring systems (ITSs), and applies it to the evaluation of the SQL-Tutor, an ITS for the database language SQL. An examination of the historical development, theory and architecture of intelligent tutoring systems, as well as the theory, architecture and behaviour of the SQL-Tutor sets the context for this study. The characteristics and criteria for evaluating computer-aided instruction (CAl) systems are considered as a background to an in-depth investigation of the characteristics and criteria appropriate for evaluating ITSs. These criteria are categorised along internal and external dimensions with the internal dimension focusing on the intrinsic features and behavioural aspects of ITSs, and the external dimension focusing on its educational impact. Several issues surrounding the evaluation of ITSs namely, approaches, methods, techniques and principles are examined, and integrated within a framework for assessing the added value of ITS technology for instructional purposes.Educational StudiesM. Sc. (Information Systems

Supporting students in the analysis of case studies for professional ethics education

Intelligent tutoring systems and computer-supported collaborative environments have been designed to enhance human learning in various domains. While a number of solid techniques have been developed in the Artificial Intelligence in Education (AIED) field to foster human learning in fundamental science domains, there is still a lack of evidence about how to support learning in so-called ill-defined domains that are characterized by the absence of formal domain theories, uncertainty about best solution strategies and teaching practices, and learners' answers represented through text and argumentation. This dissertation investigates how to support students' learning in the ill-defined domain of professional ethics through a computer-based learning system. More specifically, it examines how to support students in the analysis of case studies, which is a common pedagogical practice in the ethics domain. This dissertation describes our design considerations and a resulting system called Umka. In Umka learners analyze case studies individually and collaboratively that pose some ethical or professional dilemmas. Umka provides various types of support to learners in the analysis task. In the individual analysis it provides various kinds of feedback to arguments of learners based on predefined system knowledge. In the collaborative analysis Umka fosters learners' interactions and self-reflection through system suggestions and a specifically designed visualization. The system suggestions offer learners the chance to consider certain helpful arguments of their peers, or to interact with certain helpful peers. The visualization highlights similarities and differences between the learners' positions, and illustrates the learners' level of acceptance of each other's positions. This dissertation reports on a series of experiments in which we evaluated the effectiveness of Umka's support features, and suggests several research contributions. Through this work, it is shown that despite the ill-definedness of the ethics domain, and the consequent complications of text processing and domain modelling, it is possible to build effective tutoring systems for supporting students' learning in this domain. Moreover, the techniques developed through this research for the ethics domain can be readily expanded to other ill-defined domains, where argument, qualitative analysis, metacognition and interaction over case studies are key pedagogical practices

An Intelligent Debugging Tutor For Novice Computer Science Students

Debugging is a necessary aspect of computer science that can be difficult for novices and experienced programmers alike. This skill is mainly self-taught and is generally gained through trial and error, perhaps with some assistance from a professor or other expert figure. Novices encountering their first software defects may have few avenues open to them depending on the environment in which they are learning to program. The evident problem here is that the potential for a student to become stuck, frustrated, and/or losing confidence in their ability to pursue computer science is great. For a student to be successful when working professionally or progressing through academia they need to be able to function independently; trusting their own knowledge on par or above that of others so that their productivity does not rely on the knowledge of someone else. In order to solve this problem an Intelligent Tutoring System for teaching debugging skills to the novice utilizing Case Based Reasoning, Static Program Slicing, and the student\u27s preferred learning style was proposed. Case acquisition and automatic Exercise Generation were also explored. The system built for this research program was evaluated using novice students at the College and High School levels. Results of this evaluation produced statistically significant results at the p\u3c.05 and p\u3c.01 levels, with generated exercises exhibiting significance at the p\u3c.01 level. These results prove that the methodology chosen is a valid approach for the problem described, that the system does in fact teach students how to debug programs, and that the system is capable of successfully generating exercises on the fly