Proceedings of: First International Conference on Reforming Education, Quality of Teaching and Technology-Enhanced Learning: Learning Technologies, Quality of Education, Educational Systems, Evaluation, Pedagogies (TECH-EDUCATION 2010). Athens, Greece, May 19-21, 2010.Course evaluation is one of the key processes in the educational context that is in charge of the assurance of quality of courses taught in an institution. Although it is an essential process realized in every institution, the procedure followed for course evaluation does not comply with any evaluation standard. The objectives of this document are: to define a course evaluation concept map to understand better the needs of this field; to analyze the course evaluation standard ISO/IEC 19796, the only standard for course evaluation; and finally, to carry out a study of European institutions evaluation processes to determine why anyone is using standards in this learning process.This work was partially funded by Learn3 project, “Plan Nacional de I+D+I” TIN2008-05163/TSI and the Best Practice Network ICOPER (Grant No. ECP-2007-EDU-417007).Publicad

Crespo García, Raquel

Delgado Kloos, Carlos

Gutiérrez Rojas, Israel

Leony, Derick

Pardo Sánchez, Abelardo

Name not available

Course Evaluation Study in Europe: The Current Picture

Leony Arreaga, Derick Antonio

Universidad Carlos III de Madrid e-Archivo

This document is published in:Lytras, Miltiadis D. et al. (Eds.) (2010). Technology Enhanced Learning. Quality of Teaching and Educational Reform: First International Conference, TECH-EDUCATION 2010, Athens, Greece, May 19-21, 2010. Proceeding. (Communications in Computer and Information Science, 73) Springer, 351-356. DOI: http://dx.doi.org/10.1007/978-3-642-13166-0_50 © 2010. Springer-Verlag Berlin Heidelberg Ins t i tu t ional  Repos i tory  Course evaluation study in Europe: the current picture Israel Gutiérrez Rojas1, Raquel M. Crespo1, Derick Leony1, Abelardo Pardo1 and Carlos Delgado Kloos1,1 Universidad Carlos III de Madrid, Av. de la Universidad 30 Leganés, Madrid - Spain {igrojas, rcrespo, dleony, abel, cdk}@it.uc3m.es Abstract. Course evaluation is one of the key processes in the educational context that is in charge of the assurance of quality of courses taught in an institution. Although it is an essential process realized in every institution, the procedure followed for course evaluation does not comply with any evaluation standard. The objectives of this document are: to define a course evaluation concept map to understand better the needs of this field; to analyze the course evaluation standard ISO/IEC 19796, the only standard for course evaluation; and finally, to carry out a study of European institutions evaluation processes to determine why anyone is using standards in this learning process. Keywords: course evaluation, shared-teaching, ISO/IEC 19796 1   Introduction The main objectives of this article is to have a look at the current situation of course evaluation in European institutions; this big objective can be divided in three smaller ones: to present the development of the evaluation concept map in the domain of course evaluation; to summarize a study performed on a current quality management, assurance metrics standards; and to provide a global picture of the current status of course evaluation practices in Europe. Both theoretical models contained in this article have been developed through a series of reviewing sessions, undergoing continuous modifications in order to achieve a model capable of describing the essential elements and procedures involved in learning quality assurance. The standard studied in this paper is ISO/IEC 19796, parts 1 [1] and 3 [2] which are the only sections released up to the date of the creation of this document. These parts provide a general approach and a reference for methods and metrics within the context of quality management in information technology for learning, education and training. The current status of evaluation practices in Europe has been obtained by the evaluation of different scenarios through means such as personal interviews and 1surveys. These surveys have been filled out by some ICOPER partners in order to study the usage of course evaluation in real institutions. ICOPER [3] is an eContentPlus Best Practice Network that started its work in September 2008. As part of its objectives, ICOPER will provide a Reference Model and mechanisms to ensure European-wide user involvement, cooperation, and adoption of standards in the educational framework. To accomplish this goal, the project will systematically analyse the specifications and standards available and in use, to draw conclusions on their validity. In the context of the ICOPER project, an effort is under way to detect the course evaluation standards problems and to propose a set of best practices according to their usage in European institutions. 2   Course evaluation concepts and standards In this section the development of a concept map about course evaluation is presented. After that, and taking into account the main concepts in this field, an analysis of ISO/IEC 19796 is performed, because it is the only standard that could be used for course evaluation. 2.1   Concept map As part of the ICOPER Reference Model, a conceptual map modelling key concepts for course evaluation is being developed, by capturing key concepts and related specifications. It is important to define a set of concepts in the domain of course evaluation, on one hand to clarify the terminology used in the study presented in the next section and, on the other hand to establish the relationships between these concepts. The main focus of the evaluation concept map in Fig. 1 is centred on Course Evaluation, which is understood as the process of identifying, obtaining and interpreting data to determine which course objectives are being achieved; this definition comes from the concepts of assessment and evaluation in [4]. The Course Evaluation is ruled by a Quality Assurance Approach, usually a learning quality assurance standard, specification or guide. The data collected during the evaluation process provides a performance qualification of the unit of learning, the learning supporter and the learning assessment. The final output generated by the course evaluation is the Evaluation Result, which reports formally the quality status of the course. The ICOPER Reference Model allocates the evaluation processes within the service layer. The key processes of the evaluation domain have been identified like: -­‐ Creating survey -­‐ Visualising survey -­‐ Submitting evaluation -­‐ Visualising global results 2All of these processes belong to the evaluation stage of the process model. There has been an emphasis on the use of questionnaires to collect the evaluation data since it was observed that this was the most appropriate way to perform this task.   Fig. 1. Course evaluation concept map  2.2   ISO/IEC 19796-1 Part 1 of the ISO/IEC 19796 standard provides a general approach for quality management, assurance and metrics in learning, education and training scenarios. The purpose of [1] is to provide a Reference Framework for the Description of Quality Approaches, which is defined as a framework to describe, compare and analyze quality management and quality assurance approaches. In order to describe and elaborate this reference framework [1] includes its process model. This process model is a framework used for the description, comparison and analysis of process-oriented quality approaches and can be used in other scenarios such as the harmonization of quality approaches. The process model is divided in seven parts where every part includes a set of sub-processes or sub-aspects. It is stated in [1] that a quality description conforms to the standard if each included process corresponds to the appropriate specification and includes all sub-processes. A conforming description may contain additional processes and data elements. Due to the relevance of this framework, the ICOPER Reference Model has adopted this process model for the classification of key processes. 2.3   ISO-IEC 19796-3 Part 3 of the ISO/IEC 19796 standard [2], Reference methods and metrics, provides a harmonized description of the methods and metrics that are needed in the implementation of systems of quality management and quality assurance for 3stakeholders involved in a learning process that makes uses of information technology. Previous to classifying the methods and metrics involved in a learning quality assurance process, it is important to define these terms. In a quality approach context, method is one of a set of instruments or tools used to assure or manage quality in processes, while metric is a material measure within some aspects of quality characteristics [2]. This part of the standard provides the reference models for quality methods and for quality metrics. Some previous studies [5] explain the importance of such models and show the evolution that they experienced in order to contain the critical aspects of any quality method and quality metric. A relevant section of part 3 is the collection of methods and metrics, which consist of a classification of categories, category descriptions and subcategories of methods. 3   Course evaluation study in Europe A series of interviews and reports have been used to collect the analysed data. The participating institutions are all inside ICOPER consortium. An in depth analysis is very appropriate in this study and ICOPER consortium provide us with this possibility and may also be appropriate in reality to determine and analyse the causes of the lack of standards utilization in some of the participating organisations. The sample we have worked on has the following features that make it appropriate for the intended purpose of the article: -­‐ Deals with formal and informal learning -­‐ Deals with face-to-face, blended and pure e-learning examples -­‐ It is geographically distributed all around Europe (geographical diversity) -­‐ The sample distribution also cover multicultural and multilingual examples As shown in Table 1, studied scenarios are much heterogeneous. The commonalities between them are that the students are the course evaluators; neither teacher nor quality assurance institutions perform any evaluation. In all cases the objectives are formative (mainly for course improvement). The access to the evaluation results is quite varied, but, in general, the lecturer/instructor is the entity that has this privilege. Sometimes also the students can access to this information. The most common methodology instrument is the questionnaire/survey; some other methods like group discussion are applied but in just one case. The tools used in these institutions are mainly paper (almost all of them) with some online questionnaires, often integrated in LMS, and in other cases surveys/questionnaires are attached to specific tools. Regarding standards usage, the trend is quite clear: none of the participating institutions use a specific standard for course evaluation. An internal (ad hoc) methodology, however, is followed and institution-dependant in some cases; in other cases, evaluation management is directly conducted by the lecturer of the course. Finally, there exists a scarce use of course evaluation content repositories and the evaluation process use to be anonymous. 4Table 1.  Course evaluation survey results. Institution Evaluator Evaluee Process Repos. Artifact/Tool Other info. IMC Student Course Instructors Ad hoc Yes Survey/CLIX - JSI Student Course Tools Ad hoc No Survey/Paper - HUM Student Course Tools Ad hoc No Survey/Paper - OUNL Student Course Ad hoc No Survey /Web tool - TLU Student Course Instructors Ad hoc No Survey/Paper & Information system Anonymous ULE Student Course Ad hoc No Survey/Paper - UMU Student Course Ad hoc No Survey/phpESP, VTSurvey, LimeSurvey - WUW Student Course Ad hoc No Survey/Paper & learn@wu Survey templates AGH Student Course Ad hoc No Survey/Moodle - UNIVIE Student Course Instructors Ad hoc Yes Survey/Paper & EvaSyS, GmbH Anonymous The entities that are evaluated are the course and sometimes the instructors and tools. In the case of JSI [6], the evaluation of the course consisted on questions of several topics: educational content, assessment, communication, personalization, and directedness. The questions about tools also covered different aspects like multimedia or technical elements. In the concrete case of UK, universities are their own awarding bodies and they continually assess their systems and their courses to ensure that they are fit for purpose. In addition, all universities use a network of external experts – called external examiners - to advise on whether the standards a university sets are appropriate [8]. 3.1   Shared-teaching course evaluation The following section is about a concrete study of course evaluation in shared-teaching. It is based on a set of surveys different from the previous study. In order to capture the current state of shared-teaching evaluation in Europe, an evaluation pilot experience was performed with a group of ICOPER partners. This pilot experience was defined as the analysis and comparison of shared-teaching evaluation processes. The pilot procedure consisted of participants responding to a survey, whose topics included the evaluation of the shared-teaching scenario (evaluators, evaluees and reviewers of the evaluation results), the use of standards and the evaluation process as a whole. 5The survey responses showed the lack of use of standards for quality assurance and the use of customised quality assurance procedures to evaluate shared-teaching courses. Among the common practices mentioned by the pilot participants it could be found that the data collection is often computer based, usually through a web application. The answers provided by the evaluators are usually anonymous and the learner comments are provided to the instructors as feedback. 4   Conclusions The study detailed in this article, in special the interviews performed to the partners of ICOPER, has shown that currently there is not such thing as a course evaluation standard, since each institution manages and assures the quality of its TEL approaches using established procedures that are customised to their needs. The shared-teaching evaluation pilot experience was intended to analyse the different approaches taken to assure quality, focusing in a specific scenario. Finally, the study and summary of Parts 1 and 3 of the standard ISO/IEC 19796 are intended to serve as a guide for future competence-driven quality assurance reference models. In the course evaluation domain, a trend has been identified: none of the analysed institutions is using any standards or specifications. There appear to be some sets of guidelines used within individual institutions that are followed without any direct relation to evaluation standards. In this way, explicit quality assurance is not performed. These guidelines do not follow a common pattern easily identified, constituting a very heterogeneous set. It is fair to say that, in order to motivate organizations to go through this quality processes, references to external standards need to be emphasised, possibly by professional accrediting bodies. Acknowledgments. This work was partially funded by Learn3 project, “Plan Nacional de I+D+I” TIN2008-05163/TSI and the Best Practice Network ICOPER (Grant No. ECP-2007-EDU-417007). References 1. S. ISO, “IEC 19796-1: 2005,” Information technology-Learning, education and training-Quality management, assurance and met rics-Part 1: General approach.2. S. ISO, “IEC 19796-3: 2005,” Information technology-Learning, education and training-Quality management, assurance and metrics-Part 3: Reference methods and metrics.3. ICOPER project website. http://www.icoper.org4. ABET. 2008-2009 Accreditation Policy and Procedure Manual.5. K. Hirata, “Information Model for Quality Management Methods in e-Learning”, AdvancedLearning Technologies, 2006. Sixth International Conference on, 2006, pp. 1147- 1148.6. Bubaš, G., Balaban, I., Begičević, N. (2007). Evaluation of online courses as an element ofinstructional design: The case of two hybrid university courses. Proceedings of theConference Computers in Education, 30th International Convention MIPRO 2007, Opatija,Croatia, pp. 216-221.67. QAAHE (2009). Quality Assurance Agency for Higher Education.  http://www.qaa.ac.uk/8. Universities UK: Quality and standards in UK universities: A guide to how the systemworks, November 2008. http://www.universitiesuk.ac.uk/9. Bergström, Granberg, Segerbrant & Österlund 2006, "Course evaluation - Coursedevelopment", Journal of Teacher Education, 4, 10-32.7

http://dx.doi.org/10.1007/978-3-642-13166-0_50

Course Evaluation Study in Europe: The Current Picture

Abstract

Similar works

Full text

Available Versions

Name not available

Name not available

Universidad Carlos III de Madrid e-Archivo