Prospects for summative evaluation of CAL in higher education

Many developers and evaluators feel an external demand on them for summative evaluation of courseware. Problems soon emerge. One is that the CAL may not be used at all by students if it is not made compulsory. If one measures learning gains, how does one know that one is measuring the effect of the CAL or of the motivation in that situation? Such issues are the symptoms of the basic theoretical problem with summative evaluation, which is that CAL does not cause learning like turning on a tap, any more than a book does. Instead, it is one rather small factor in a complex situation. It is of course possible to do highly controlled experiments: for example to motivate the subjects in a standardized way. This should lead to measurements that are repeatable by other similar experiments. However they will be measurements that have little power to predict the outcome when the CAL is used in real courses. Hence the simple view of summative evaluation must be abandoned. Yet it is possible to gather useful information by studying how a piece of CAL is used in a real course and what the outcomes were. Although this does not guarantee the same outcomes for another purchaser, it is obviously useful to know that the CAL has been used successfully one or more times, and how it was used on those occasions. Such studies can also serve a different ‘integrative’ rather than summative function by pointing out failings of the CAL software and suggesting how to remedy them

An evaluation of learning resources in the teaching of formal philosophical methods

In any discipline, across a wide variety of subjects, there are numerous learning resources available to students. For many students the resources that will be most beneficial to them are quickly apparent but, because of the nature of philosophy and the philosophical method, it is not immediately clear which resources will be most valuable to students for whom the development of critical thinking skills is crucial. If we are to support these students effectively in their learning we must establish what these resources are how we can continue to maintain and improve them, and how we can encourage students to make good use of them. In this paper we describe and assess our evaluation of the use made by students of learning resources in the context of learning logic and in developing their critical thinking skills. We also assess the use of a new resource, electronic handsets, the purpose of which is to encourage students to respond to questions in lectures and to gain feedback about how they are progressing with the material

An exploration of tools, techniques and procedures for evaluating informational/educational multimedia software

This thesis explored the evaluation of informational/educational systems. It investigated the features of the design of an evaluation system needed to conduct the summative evaluation of informational and educational software. In designing this evaluation methodology, several questions were addressed: - What is needed to be achieved from an evaluation? - Which characteristics/features of a system required evaluation? - How was the evaluation going to be conducted? - How can future developers use the information obtained from the evaluation? - What were the scope and the limitations of the proposed evaluation? - How can evaluation tools, techniques and procedures developed by this research, be useful in the design of evaluation systems for similar informational/educational software? To answer these questions an in-depth review of the available literature and electronic resources that address the area of multimedia software evaluation was conducted. In order to identify the best tools to conduct an efficient and effective evaluation it was also imperative to examine and assess tools and techniques already developed. It was then possible to proceed with the design and construction of an evaluation system, based on a thorough familiarity with and understanding of previous work conducted in this area. The EduKit2000 CD produced and distributed by Edith Cowan University to all commencing and external students was selected as a case study with which to implement and test the evaluation system. Because of the availability and the proximity of both the developers and the end-users, it was considered an appropriate choice. The evaluation program developed involved the use of four evaluation tools. These consisted of an expert review, a questionnaire, user and novice testing. The questionnaire was sent out with the CD. The feedback received from the questionnaire was analysed in conjunction with the data returned from the other evaluation tools and the summative evaluation of the CD was conducted, in order to determine whether the product reflected the requirements of its developers. Analysis of the returned research data aimed to discover what future changes may be required to be implemented to the original product in order for it to fulfil those requirements, while still remaining a cost-effective, freely distributed product. More importantly, this evaluation of EduKit2000 was intended as a road test of the methodology developed by this research. It made it possible to identify and improve some features of the evaluation plan and strategy that did not perform us well as anticipated. The research conducted for the purpose of this thesis explored the evaluation of informational systems and achieved four distinct objectives: 1. A methodology was researched, designed and developed to comprehensively evaluate informational/educational systems 2. This methodology was tested by using a case study (EduKit2000) to which it was applied. 3. The methodology was evaluated and reviewed, and recommendations were formulated for improvements. 4. A comprehensive evaluation of EduKit2000 was produced, along with recommendations for improving the product. By adequately identifying those areas that can be improved in order for the product to fulfil its objectives, the evaluation system developed by this research was demonstrated to be an effective, and cost-effective, methodology for evaluations of informational/educational software

Metadata quality issues in learning repositories

Metadata lies at the heart of every digital repository project in the sense that it defines and drives the description of digital content stored in the repositories. Metadata allows content to be successfully stored, managed and retrieved but also preserved in the long-term. Despite the enormous importance of metadata in digital repositories, one that is widely recognized, studies indicate that what is defined as metadata quality, is relatively low in most cases of digital repositories. Metadata quality is loosely defined as "fitness for purpose" meaning that low quality of metadata means that metadata cannot fulfill its purpose which is to allow for the successful storage, management and retrieval of resources. In practice, low metadata quality leads to ineffective searches for content, ones that recall the wrong resources or even worse, no resources which makes them invisible to the intended user, that is the "client" of each digital repository. The present dissertation approaches this problem by proposing a comprehensive metadata quality assurance method, namely the Metadata Quality Assurance Certification Process (MQACP). The basic idea of this dissertation is to propose a set of methods that can be deployed throughout the lifecycle of a repository to ensure that metadata generated from content providers are of high quality. These methods have to be straightforward, simple to apply with measurable results. They also have to be adaptable with minimum effort so that they can be used in different contexts easily. This set of methods was described analytically, taking into account the actors needed to apply them, describing the tools needed and defining the anticipated outcomes. In order to test our proposal, we applied it on a Learning Federation of repositories, from day 1 of its existence until it reached its maturity and regular operation. We supported the metadata creation process throughout the different phases of the repositories involved by setting up specific experiments using the methods and tools of the MQACP. Throughout each phase, we measured the resulting metadata quality to certify that the anticipated improvement in metadata quality actually took place. Lastly, through these different phases, the cost of the MQACP application was measured to provide a comparison basis for future applications. Based on the success of this first application, we decided to validate the MQACP approach by applying it on another two cases of a Cultural and a Research Federation of repositories. This would allow us to prove the transferability of the approach to other cases the present some similarities with the initial one but mainly significant differences. The results showed that the MQACP was successfully adapted to the new contexts, with minimum adaptations needed, with similar results produced and also with comparable costs. In addition, looking closer at the common experiments carried out in each phase of each use case, we were able to identify interesting patterns in the behavior of content providers that can be further researched. The dissertation is completed with a set of future research directions that came out of the cases examined. These research directions can be explored in order to support the next version of the MQACP in terms of the methods deployed, the tools used to assess metadata quality as well as the cost analysis of the MQACP methods

Metadata quality issues in learning repositories

Metadata lies at the heart of every digital repository project in the sense that it defines and drives the description of digital content stored in the repositories. Metadata allows content to be successfully stored, managed and retrieved but also preserved in the long-term. Despite the enormous importance of metadata in digital repositories, one that is widely recognized, studies indicate that what is defined as metadata quality, is relatively low in most cases of digital repositories. Metadata quality is loosely defined as "fitness for purpose" meaning that low quality of metadata means that metadata cannot fulfill its purpose which is to allow for the successful storage, management and retrieval of resources. In practice, low metadata quality leads to ineffective searches for content, ones that recall the wrong resources or even worse, no resources which makes them invisible to the intended user, that is the "client" of each digital repository. The present dissertation approaches this problem by proposing a comprehensive metadata quality assurance method, namely the Metadata Quality Assurance Certification Process (MQACP). The basic idea of this dissertation is to propose a set of methods that can be deployed throughout the lifecycle of a repository to ensure that metadata generated from content providers are of high quality. These methods have to be straightforward, simple to apply with measurable results. They also have to be adaptable with minimum effort so that they can be used in different contexts easily. This set of methods was described analytically, taking into account the actors needed to apply them, describing the tools needed and defining the anticipated outcomes. In order to test our proposal, we applied it on a Learning Federation of repositories, from day 1 of its existence until it reached its maturity and regular operation. We supported the metadata creation process throughout the different phases of the repositories involved by setting up specific experiments using the methods and tools of the MQACP. Throughout each phase, we measured the resulting metadata quality to certify that the anticipated improvement in metadata quality actually took place. Lastly, through these different phases, the cost of the MQACP application was measured to provide a comparison basis for future applications. Based on the success of this first application, we decided to validate the MQACP approach by applying it on another two cases of a Cultural and a Research Federation of repositories. This would allow us to prove the transferability of the approach to other cases the present some similarities with the initial one but mainly significant differences. The results showed that the MQACP was successfully adapted to the new contexts, with minimum adaptations needed, with similar results produced and also with comparable costs. In addition, looking closer at the common experiments carried out in each phase of each use case, we were able to identify interesting patterns in the behavior of content providers that can be further researched. The dissertation is completed with a set of future research directions that came out of the cases examined. These research directions can be explored in order to support the next version of the MQACP in terms of the methods deployed, the tools used to assess metadata quality as well as the cost analysis of the MQACP methods

Full Report: Use and Users of Digital Resources: A Focus on Undergraduate Education in the Humanities and Social Sciences

A "build it and they will come" approach to many university digitization initiatives has precluded systematic investigations of the demand for these resources. Those who fund and develop digital resources have identified the general lack of knowledge about the level and quality of their use in educational settings as pressing concerns. This full report describes our research and the complete results of a large 2-year study. The purpose of our research was (1) to map the universe of digital resources available to a subset of undergraduate educators in the humanities and social sciences, and (2) to investigate how and if available digital resources are actually being used in undergraduate teaching environments. We employed multiple methods, including surveys and focus groups. Our definition of digital resources was intentionally broad and included rich media objects (e.g., maps, video, images, etc.) as well as text

An evaluation of factors affecting students' use of a web-based engineering resource

The purpose of this work was to investigate the relationship between a number of influential factors, including cognitive style and approach to learning, and students’ processing behaviour during their use of a particular Web-based resource for Electronics and Electrical Engineering undergraduates. This was achieved through the development of a learner profile for each student using Riding’s (1991) Revised Study Process Questionnaire (R-SPQ-2F). The quantitative component of the research was then set against a detailed analysis of students’ processing behaviour using verbal protocol data gathered through individual think-aloud sessions and post-intervention interviews. The results of the quantitative component of the research provided no compelling evidence to suggest that cognitive style was a factor that influenced student performance while using the resource or their perceptions of the package. There was however, some evidence to suggest that the package was more positively received by students who profiled as deep learners than their surface counterparts. The analysis of students’ processing behaviour from their verbal protocols highlighted a number of the resource’s shortcomings, which typically promoted a surface, goal-oriented approach to its content. It also identified problems with the design and structure of the resource, which at times had a deleterious effect on learning. The results also raised questions regarding the efficacy and use of psychometric inventories in this kind of research

Integrative evaluation of computer assisted learning in geography in schools and university

Evaluating CAL as part of an overall teaching and learning situation, can help school and university teachers to recognise strengths and weaknesses in their use and delivery of teaching method. This approach is called "integrative evaluation". This research extends the application of integrative evaluation methods to Computer Assisted Learning (CAL) in three ways: (1) to use the integrative approach at the secondary school level for the first time; (2) to investigate deep vs. surface learning in university students using CAL in conjunction with other resources; (3) to apply integrative evaluation to the field of CAL in Geography teaching for the first time. The research comprises four studies, each dealing with a different CAL package. Two of the studies were conducted in two secondary schools in the city of Glasgow, and two were carried out in the Geography Department at the University of Glasgow. The total sample population was 238 (74 school pupils and 164 university students). Various instruments besides classroom observation were used for evaluation purposes, including tests designed to measure learning outcomes, questionnaires designed to gauge pupils'/students' reactions, opinions and confidence levels and interviews. The software studied in this research comprised: (1) A database-like dealing with the geography of Japan. (2) An interactive CAL package related to the subject of the Weather. (3) A university-level CAL package on the subject of Glaciation which related theory to practical labs. (4) Five different application programs (including GIS and Minitab) being taught to university students in an IT course. In the 3 studies that measured learning, the evaluations showed definite gain due to the CAL, although with interesting variations from objective to objective. Among the other findings of the research are: (1) Two distinct patterns of correlation between CAL gains and pupils' geographical ability were detected in each of the school-based studies. It was concluded that low ability pupils gained more from a stimulating, interactive CAL package, but benefited less than more able pupils from a database-type package. (2) Students found the scheduled computer lab the most useful learning resource for learning about the five applications covered in the IT course. This finding clearly demonstrated that each resource had a specific role to play within the course, and that each resource is more suited to the achievement of certain learning objectives and less suitable for others. (3) Only a small shift from surface to deep learning was found by the end of the Glaciation course, even though the CAL package was specifically designed to link theoretical and practical knowledge. However, doubts raised about the design of questions used to determine the shift suggest that the shift may have been underestimated