It is challenging to enhance quality of e-learning. We based the quality assessment on the EFQM and the Kirkpatrick models. We built our own SEVAQ quality model on which we based our multi-functional self-evaluation questionnaire. We focused and limited ourselves on the valuable learners feedback. Our questionnaire is structured around  the 3 main criteria: the enabling learning resources, learning processes and the learning results. Questions are linked with their criteria and subcriteria. 
The SEVAQ tool addresses the designer of the questionnaire as well as the learner, being the evaluator and so being the user of the questionnaire. The designer selects on a flexible way a set of relevant questions to be answered by the learner. After finishing the questionnaire by the learner, the system analysis the result

Schreurs, Jeanne

Online-Journals.org (International Association of Online Engineering)

SEVAQ: A UNIQUE MULTI-FUNCTIONAL TOOL FOR ASSESSING AND IMPROVING THE QUALITY OF E-COURSES  SEVAQ: A Unique Multi-functional Tool for Assessing and Improving the Quality of e-Courses J. Schreurs1, A.M.Husson2, B.Merison3, E.Morin4, H. Van Heysbroeck5 1 Hasselt University, Belgium, 2 le Preau/CCIP, France 3 BLA, Bristish Learning Association, UK 4 Vidéoscope, France, 5 EFODL, Belgium    Abstract—It is challenging to enhance quality of e-learning. We based the quality assessment on the EFQM and the Kirkpatrick models. We built our own SEVAQ quality model on which we based our multi-functional self-evaluation questionnaire. We focused and limited ourselves on the valuable learners feedback. Our questionnaire is structured around  the 3 main criteria: the enabling learning resources, learning processes and the learning results. Questions are linked with their criteria and subcriteria.  The SEVAQ tool addresses the designer of the questionnaire as well as the learner, being the evaluator and so being the user of the questionnaire. The designer selects on a flexible way a set of relevant questions to be answered by the learner. After finishing the questionnaire by the learner, the system analysis the results Index Terms—quality management, TQM, e-learning services, EFQM, Kirkpatrick, self evaluation I. INTRODUCTION It is challenging to support and enhance quality management in e-learning. The need for quality management in e-learning has risen since the use of e-learning has expanded. We have to find answers on the following questions: how is quality defined, How can quality be assessed?  The same principles apply to quality of e-learning  as in the quality of teaching and learning in general. However, there are some special characteristics in e-learning that need to be specified. Quality management has to cover the teaching and learning, including the organization of the learning process, the learning content and the pedagogical and technical support for e-learning including equipment and facilities. Quality criteria have to be defined for those areas. II. THE EFQM EXCELLENCE MODEL The EFQM Excellence Model is a practical tool to help organizations do this by measuring where they are on the path to excellence, helping them understand the gaps and then stimulating solutions. The EFQM Excellence Model is a non-prescriptive framework based on 9 criteria. Five of these are 'Enablers' and four are 'Results'. The 'Enabler' criteria cover what an organization does. The 'Results' criteria cover what an organization achieves. 'Results' are caused by 'Enablers' and 'Enablers' are improved using feedback from 'Results'. Each of the nine criteria is supported by a number of  sub-criteria. Those pose a number of questions that should be considered in the course of an assessment. For example in the criteria leadership, focus is on the way management can motivate and stimulate the organization to evolve to continuous improvement. Some questions: How is management engaged in creating a culture of continuous improvement? How is management supporting the improvement activities? How is management evaluating and motivating the staff? The EFQM Model is presented in diagram form below. The arrows emphasize the dynamic nature of the Model. They show innovation and learning helping to improve enablers that in turn lead to improved results  Figure 1.   The EFQM excellence model III. KIRKPATRICK E-LEARNING EVALUATION MODEL D. Kirkpatrick presented a four-level model of quality assessment, that can be applied to traditional way of learning and also to e-learning.  1. Students’ reaction: students are asked to evaluate the training after completing the program. First is asked how well they like the training. But other questions are about the relevance of and the fitting to the objectives, the quality of the included interactive exercises, the ease of navigation, …  2. Learning results: has the learner increased his knowledge of the topic? What about the achievement? iJET – Volume 3, Issue 1, March 2008 61SEVAQ: A UNIQUE MULTI-FUNCTIONAL TOOL FOR ASSESSING AND IMPROVING THE QUALITY OF E-COURSES  3. Impact of learning on the functioning in the workplace: Are any of the new knowledge and skills retained and transferred back on the job? Is the student’s behaviour changed as a result of new learning? 4. Impact of learning on the business results: the evaluation of the business impact of the training must be measured. IV. THE SEVAQ ASSESSMENT MODEL SEVAQ stands for Self  Evaluation of  Quality in e-Learning The main goal of the SEVAQ project is improving the quality of the vocational and educational courses that are offered through open-and distance learning, e-learning and blended learning . A new multi-functional self-evaluation questionnaire has been developed in order to obtain valuable learners feedback. It is based on the EFQM-model (figure 2). The validation by the Target Reference Groups (TRGs) is ongoing now. These TRGs have been involved from the beginning of the project in order to reach the different actors from different sectors such as small and medium sized businesses, representatives of risk groups, training providers, schools, policy makers, … A. Premises of the SEVAQ evaluation model The SEVAQ assessment tool is based on the EFQM management (TQM) concept. All actors and stakeholders have to participate in the evaluation.   But in learning services in most cases evaluation will be limited to the learners evaluation task and so the TQM model will be limited to a subset of criteria, namely those that can be measured by the learner. In that case, the evaluation itself is limited to a subset of the EFQM criteria domains. Some examples are the learning content, the learning process and the resources on point of infrastructure and organisation of the learning process activities. B. Simplified e-learning EFQM model We focus on the learning/training organization or more specific on the learning department of a company or an institute, that is organizing the learning activities and also the e-learning activities for the staff of all the other company departments.  To become excellent, the learning department has to balance and satisfy the needs of all relevant stakeholders. On the enabler side the stakeholders are the company management and the management of the learning department. On the results side the stakeholders are the company departments to which the learners belong, the individual learners, and the society and economic environment. The suppliers and the customers of the company arealso involved in an indirect way because training will have an impact on the product quality. We developed a simplified e-learning EFQM model,   From the enabler site of the EFQM model, we dropped the management level. It is impossible that the learner, being an internal staffmember being trained or an external student can evaluate the management of the company.  Figure 2.  The stakeholder model for the e-learning activity We limited the model to the processes and resources. On the results site we limited the model to the  learner. All other stakeholders could be asked to evaluate the results too. But in the frame of this project we limited our  evaluation tool to learners evaluation. The limited EFQM model applied for the learning activity can be found in figure 3. The two rows are presenting the enablers and the results. In the first column we see the process of development and acquisition of the resources to be used in the learning process. In the second column we see the resources themselves being the infrastructure needed for the organisation of the learning process and the learning materials.  The third column is devoted to the learning process.  Our tool must be useful for the evaluation of the individual e-learning activity and for the classroom activities, both being part of the blended learning process we have in mind. Finally special attention goes to the support and to the control processes of the learner in the learning process.  Figure 3.  Quality criteria model, based on the EFQM excellence- and the Kirkpatrick’s evaluation model We see that the Kirckpatrick model has been integrated in our model too in the following way:    62 http://www.i-jet.orgSEVAQ: A UNIQUE MULTI-FUNCTIONAL TOOL FOR ASSESSING AND IMPROVING THE QUALITY OF E-COURSES   “learner reaction”  covers the evaluation by the learner of the characteristics of all enablers, resources as well as processes         ”learning results”  covers the effects on personal learning of the learner          ”learning impact” covers the effects of learning on the functioning of the learner in the job, and the results for the company  In this EFQM-Kirckpatrick model we have to remove also two criteria domains, where the learner can not be the evaluator.  The process of development and acquisition from one side and the learning impact from the other side can not be evaluated by the learner. V. THE SEVAQ QUESTIONNAIRE  Following our concept, we developed our questionnaire in a 3 level structure: 3 main criteria, each containing more criteria which consist of more subcriteria. Questions are linked with those subcriteria. An example can be seen in Figure 4 and Figure 5  Enabling Learning Processes 1. Organisation of services 1.1. Guidance in the choice and selection of your course 1.2. Registration process 1.3. Welcome 1.4. Organisation services and administration 2. E-learning activities 2.1. Course progression 2.2. Teaching approach/Course design 2.3. Personalising the learner's e-learning course Figure 4.   List of criteria  E-learning activities Course progression You were able to read through the course content easily, as it was presented in a logical way. You could read through the course content at your convenience. It was possible to access additional learning resources, as required. Supporting facilities such as a glossary or calendar were available to use. A range of communication facilities were available to use. *Uploading pages was effective and easy to use. You were able to read a summary of the course. You were able to navigate back to previous pages easily, if required. You were able to skip past pages easily, if required. You were able to complete practice exercises and self-assessments, as required. Figure 5.  List of questions linked with sub-criteria 2.1 VI. THE SEVAQ TOOL SEVAQ is a self-evaluation tool of Quality in e-learning. The tool addresses the designer of the questionnaire as well as the learner, being the evaluator and so being the user of the questionnaire. The system also differentiates between an individual learner, known by the system and a group of learners not identified individually. The designer selects on a flexible way a set of relevant questions to be answered by the learner. The selection can be done on criterion , on sub-criterion or on question level. (figure 7 and figure 8) Dependent of the level of the learner the designer decides on a short or a long answering method. In the long answering method, the learner is asked about the necessity of improving the resources or the processes.   Figure 6.  The roles of different users: the designer area  iJET – Volume 3, Issue 1, March 2008 63SEVAQ: A UNIQUE MULTI-FUNCTIONAL TOOL FOR ASSESSING AND IMPROVING THE QUALITY OF E-COURSES  After finishing the questionnaire by the learner, the system analysis the results of the individual learners and on group level. The designer of the course can evaluate the course and as a consequence improve it on those points being evaluated as weak by the learners.   Figure 7.   A questionnaire based on a three level structure  Figure 8.   Selection of questions on criterion and subcriteria level VII. CONCLUSION We built our own SEVAQ quality model based on EFQM and Kirkpatrick.  We developed our multi-functional self-evaluation questionnaire. We focused and limited ourselves on the valuable learners feedback. Our questionnaire is structured around  the 3 main criteria: the enabling learning resources, learning processes and the learning results. Questions are linked with their criteria and subcriteria.  The SEVAQ tool addresses the designer of the questionnaire as well as the learner, being the evaluator and so being the user of the questionnaire. The designer selects on a flexible way a set of relevant questions to be answered by the learner. After finishing the questionnaire by the learner, the system analysis the results. The organisation of learning services must be seen as a business activity and also evaluated in that way. We can advice that the evaluation has to be extended as regards content as well as evaluators. The full EFQM model can be applied and our SEVAQ quality model can be adjusted in that way. REFERENCES [1] Jeanne Schreurs, Rachel.moreau: "The EFQM self-assessment model in e-learning quality management." In Proceedings of Conference EDEN2006 Wenen 15-17 June 2006. Pages 233-238; ISBN 963 06 0063 3 [2] Jeanne Schreurs, Rachel Moreau: EFQM and Kirkpatrick in a Framework for Evaluation of E-learning" in Proceedings of the World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, E-Learn 2006, to be held in Honolulu, Hawaii, USA, October 13-17, 2006. page 1780-1785. AACE, ISBN 1-880094-60-6 [3] A.M.Husson, B.Merison, J.Schreurs, E.Morin, H.Van Heysbroeck: "European self-evaluation tool for e-learning: an ongoing focus on quality and customer's needs" in Proceedings of the 11th Int. Conf. On technology supported Learning& Training: Online EDUCA Berlin Nov29-Dec 2, 2006. page 466-469; ISBN 3-9810562-3-X   AUTHOR J. Schreurs is Professor at the Faculteit Toegepaste Economische Wetenschappen, Universiteit Hasselt, Campus Diepenbeek, Agoralaan gebouw D, 3590 Diepenbeek, Belgium.  (e-mail: Jeanne.Schreurs@uhasselt.be).   Manuscript received 03 September 2007 Published as submitted by the author.  64 http://www.i-jet.org

SEVAQ: a unique multi-functional tool for assessing and improving the quality of e-courses

Abstract

Similar works

Full text

Available Versions

Online-Journals.org (International Association of Online Engineering)