Skip to main content
Article thumbnail
Location of Repository

A study of methods of evaluating multimedia materials for language learning

By Saima Nawaz Sherazi


There is a long tradition of paper based materials evaluation in ELT, but at this juncture, a scarcity of studies on ELT Multimedia (MM) materials evaluation. Such studies as have been undertaken have tended to adopt the perspective of the materials developer rather than the end user. But there have been no developed studies of evaluation methods which could be adopted by potential users. Despite calls being made for systematic evaluation, not many proposals have been developed, and there has been little exploration of potential best practice, or of the 'goodness of fit' between methods and evaluation puposes. This study aims to investigate evaluation methods in order to establish best practice in the evaluation of multimedia CALL applications, with a focus on learners' experience and opinions, and with the aim of enabling potential teacher-users of CALL materials to gauge the suitability of materials for their learners. Chapter one of this thesis provides a rationale for this study and an overview of the background to this research. Chapter two presents a review of literature undertaken in four domains: educational evaluation and research methods; Human Computer Interaction (HCI) usability evaluation methods; ELT materials evaluation; and studies ofCALL materials in use. Chapter three focuses on the design and conduct of the study by explaining how some methods of evaluation were trialled in a pilot study and four were selected for adoption and analysis in the main study. The methods selected were Foeu.r Groups, Rttrospective Protorolr, PLUM and SUM! Questionnaires and Activity Monitonitg. To determine the qualities and limitations of these methods, a set of criteria was developed from the literature on software usability evaluation methods in HCI and a broader literature on educational evaluation and research methods. The four data study chapters (4-7) each discuss one of the chosen methods and descnbe how the method was ,6perationalised in an evaluation of learner responses to multimedia software. The final chapter draws together the discussion of the findings and presents different proposals for best practice. The focus in the discussion of findings is on how the chosen methods performed according to the set of criteria. The findings confirm that focus groups and questionnaires are quick and efficient methods whereas retrospective protocols and activity monitoring provide more detailed and protracted data. Teacher evaluators can be guided by the objectives of their evaluation to explore different combinations ofthese methods. Participants in the pilot were 12 ESL students from the University of Warwick and in the main study 45 Freshman/sophomore students from a university in Pakistan. The materials used to operationalise methods were the EASE CD-ROMS listening to Lectures and Seminar Skills 1: Presentations. The research contributes to the field in undertaking an in-depth and extensive study of evaluation methods applicable to CALL materials, whi~ adopt a leamer-centred perspective, and conform to sound principles within educational evaluation, yet which draw on practice in the field of HCI, since this expertise is so relevant in the rapid development of multimedia materials for use in ELT. Moreover, by developing the composite set of core criteria this study has created a tool which practitioners in the field can use to select most appropriate methods for their particular evaluation purposes

Topics: LB
OAI identifier:

Suggested articles


  1. (2000). 2' Ed. ManoulatingMeetings. How to get whatyou want whenyou want it. Trans-Atlantic Pubns.
  2. (2003). 5th Ed. Research Methods in Education. London,
  3. 9241-11(1995): Ergonomic Requirements for Office Work with Visual Display Terminals, Part 11, Guidance on specifying and measuring usability, doi
  4. 9241-11(1999): Ergonomic Requirements for Office Work with Visual Display Terminals, Part 11, Guidance on specifying and measuring urabi/ity,
  5. 993)Insiders and Outsiders: Exploring Ethnocentrism and Cultural Relativity in Sociology Courses. doi
  6. (1994). A doi
  7. (2001). A (2001)The State of the Art in Automating Usability Evaluation of User Interfaces. doi
  8. (1990). A context adaptive model for program evaluation. doi
  9. (1996). A Course in Language Teaching, Cambridge: doi
  10. (1988). A criterion-based approach to software evaluation. doi
  11. (2003). A critique of using grounded theory as a research method.
  12. (1991). A framework for human factors evaluation. doi
  13. (1992). A methodological framework for CALL courseware development
  14. (1993). A practical guide to usability testing,
  15. (1995). A review of the types of CALL programs for vocabulary instruction. doi
  16. (1996). A theoretical foundation for the development of pedagogical tasks in computer mediated communication.
  17. (1975). An Introduction to Curriculum Research and Development.
  18. (1983). Appendix: Evaluating Materials',
  19. (1986). Approaches and Methods in Language Teaching, Cambridge: doi
  20. (1989). Appropriate design: the internal organisation of course units', doi
  21. (2006). BAAL(1994) Recommendations on Good practice in Applied Lnguistics available at http: //www.
  22. (1993). Bread and dreams or bread and circuses? A critique of `case study' research in education.
  23. (2002). CALL and the classroom: the case for comparative research, doi
  24. (1990). CALL in use-use of CALL: doi
  25. (1999). Care Study Research in EducationalSettings.
  26. (1993). Case Studies of Navigational Patterns in Constructive Hypertext', doi
  27. (2000). Choosing CALL software: beginning the evaluation process, TESOLjournal 9,18-22 Bassey,
  28. (1991). Choosing the best available textbook. '
  29. (1995). Choosing Your Coursebook,
  30. (1951). Coefficient alpha and the internal structure of tests. doi
  31. (1992). Cognitive Walkthroughs: A Method for TheoryBased Evaluation of User Interfaces', doi
  32. (1998). Comparative evaluation of usability tests.
  33. (2004). Computer-mediated collaborative learning. doi
  34. (1997). ComputerAssisted Language Learning: Context and Conceptualisation.
  35. (1985). Computern, Language Learning, and Language Teaching. Cambridge: doi
  36. (1995). Computers in English Language Learning. doi
  37. (1996). Context and Consciousness. "Activity Theory and Human-Computer Interaction. doi
  38. (1963). Course improvement through evaluation. Teachers College Record, doi
  39. (1990). Currents of Change in English Language Teaching Oxford:
  40. (1983). Curriculum Studies and Educational Planning. London: Hodder and Stoughton. doi
  41. (2003). Design and Evaluation of an Online Test assessment conceived as a complementary CALL tool. doi
  42. (2006). Design Issues Related to the Evaluation of Learner - Computer Interaction in a Webbased Environment: Activities v. doi
  43. (2001). Design, validation, and use of an evaluation instrument for monitoring systemic reform. doi
  44. (1983). Developing criteria for textbook evaluation', doi
  45. (1990). Developmental evaluations of software for young children.
  46. (1996). Different tasks, different processes? An investigation of revision behanior on two academic tasks". Conference Proceedings Second Language Acquis Acquisition and writing: A multidisciplinary approach.
  47. (1995). Discount video analysis for usability engineering. In doi
  48. (1991). Dual Coding Theory and Education. Educational Pychology Review, doi
  49. (1981). Effective evaluation: Improving the usefulness of evaluation results through responsive and naturalistic approaches. doi
  50. (2004). Effects of Online Academic Lectures On ESL listening Comprrehension, Incidental Vocabulary Acquisition, and Stategy Use. ComputerArrirted Language Learning. doi
  51. (1996). Elements of CALL methodology: Development, Evaluation, and Implementation.
  52. (2002). Elicitation techniques for interviewing. In doi
  53. (1987). Englch forSpecfic Purposes a Learning-Centred Approach, Cambridge: doi
  54. (2004). Enhancing Online CALL Design: the case for evaluation. doi
  55. (1991). Ergonomic Requirements for Office Work with Visual Display Terminals, Part 10, Dialogue Principles Draft International Standard. doi
  56. (1972). Evaluating and adapting language materials',
  57. (1975). Evaluating beginning coursebooks. English Language Teaching Forum XIII/3/4,
  58. (1978). Evaluating Beginning textbooks' reprinted as Appendix 3 in H. Madsen and j Bowen(1978), Adaptation in Language Teaching,
  59. (1994). Evaluating computer assisted language learning from the learners' point of view. In Graddol, D and Swann, j (eds) Evaluating Language: Papers fmm the Annual Meeting of the British Assn ation forApplied linguistics Held at the
  60. (1986). Evaluating educational programmes. In
  61. (1990). Evaluating instructional software. doi
  62. (1993). Evaluating Multimedia Applications, Computers and Education, doi
  63. (1994). Evaluation and English language teaching. State-of-the-art article', doi
  64. (1976). Evaluation and the control of education. In D. Tawney (ed) Curriculum Evaluation Today: Trends and Implications.
  65. (1976). Evaluation as illumination: A new approach to the study of innovative programmes.
  66. (1996). Evaluation: turning technology f vm toy to tool Report of the Working Group on Evaluation. Integrating Tech. into CSE doi
  67. (2004). Evolution of educational software evaluation: instructional software assessment.
  68. (1984). Experiential Learning, doi
  69. (1994). Focus Getup :a Practical Guide forApplied Research. 2'" Ed. Thousand Oaks Sage.
  70. (2000). Focus Getups A Practical Guide forApplied Research. Thousand Oaks,
  71. (1989). Focus Group Interviews. A Reader. 2nd Ed.
  72. (2004). Focus Group Practice. London: Sage Quentin-Baxter M
  73. (2001). Focus groups (Third
  74. (1997). Focus Groups arQualitative Research. 2"d edition. London, Thousand Oaks CA: Sage 280 ýý
  75. (1988). Focus groups as qualitative research. doi
  76. (1988). Focus groups.. a practicalguide for applied research.
  77. (1997). Focus Groups', Social Research Update, Issue Nineteen. Department of Sociology. University of Surrey. Available at //www.
  78. (1992). Focusgroups: theory and practice.
  79. (1988). Formative and summative evaluation of instructional products and learners.
  80. (1994). Fourth Generation Evaluation, doi
  81. (2000). Frameworks for the Evaluation of Multimedia Learning Technologies: Lessons learned and future directions? ' Pragramme on Learner Use of Media (PLUM)
  82. (1987). From product to process: Introspective methods in second language research". Fa: rch and Kasper
  83. (1973). Goal-free evaluation. In doi
  84. (2002). Grounded Theory- A Practical Guide for Management, Business and Market Researchers.
  85. (1998). Grounded Theory: the missing methodology on the interpretivist agenda. Qualitative Marketing Research: An doi
  86. (1990). Heuristic evaluation of user interfaces, doi
  87. (1994). Heuristic evaluation. In
  88. (2000). How to evaluate educational software,
  89. (1994). Human-Computer Interaction, Workingham, doi
  90. (1995). Hypertext, hypermedia, multimedia defined? doi
  91. (1994). I'he textbook as agent of change',
  92. (1971). Imagery and Verbal Processes, doi
  93. (1984). Improving the Quality and effectiveness of computermediated instruction through usability evaluations. ' doi
  94. (1994). Incorporating aspects of style and tone in self-access CALL courseware'
  95. (2001). Incorporating ESOL learners' feedback and usability testing in instructor developed CALL materials, TESOLjournal,
  96. (1996). Influences on course revision: an RAP case study',
  97. (2000). Integrating multimedia into the Business English curriculum: a casestudy. English forspeciic purposes 19,269-290 doi
  98. (1994). Interactive Multimedia in University Education: Designing for Change in Teaching and Learning (A-59) In
  99. (2001). Interpreting Qualitative Data -MethodsforAnalyring Talk, Text and Interaction, doi
  100. (2002). Interviewing in Education In doi
  101. (1995). Introducing focus groups', doi
  102. (1978). Introduction to factor analysis: What it is and how to do it. Thousand Oaks, CA: Sage Publications, Quantitative Appliications in the Social Sciences Series,
  103. (1990). Issues in qualitative research',
  104. (2000). Key factors in the Design of an Interactive Multimedia CD-Rom for EAP learning.
  105. (2003). Language learning online: Towards best practice. Lisse: Swets
  106. (2000). Learner autonomy and ICT: a web-based course for psychology, doi
  107. (2003). Learners' Behaviors in Computer-Based Input Activities Elicited Through Tracking Technologies, doi
  108. (2000). Learning about Learning: nsourrcesfor promoting effective learning, doi
  109. (1987). Making the Most of Your Textbook,
  110. (2006). Marketing Research, An Applied Orientation, 3rd European Edition,
  111. (1991). Marketing Research: Methodological Foundations. doi
  112. (1998). Materials Development in language Teaching, Cambridge: doi
  113. (2002). Materials Evaluation and Deezgn forLsnguage Teaching. Edinburgh Textbooks in Applied Linguistics.
  114. (1986). Mental Representations ADual-CodingAppivach, doi
  115. (1981). Metacognition in reading and writing. The development and facilitation of selective attention strategies for learning from texts.
  116. (1982). Method: Approach, design, and procedure. doi
  117. (1980). Motivation, leadership, and organization: Do American theories apply abroad. doi
  118. (1995). Multimedia and Hypertext. The Internet and Beyond. doi
  119. (1993). Multimedia: Making it Work. doi
  120. (1985). Naturalistic Inquiry. doi
  121. (1989). Nick Hammond and Lesley Allinson. "Extending Hypertext for Learning: An Investigation of Access and Guidance Tools"
  122. (2000). Online strategies for teaching thinking.
  123. (1972). Originally printed in
  124. (2000). Paradigmatic controversies, contradictions and emerging confluences'
  125. (1982). Perceiving and pursuing learners; needs',
  126. (1999). Predicting quality in educational software: Evaluating for learning, usability and the synergy between them.
  127. (1993). Protocol analysis. Verbal reports as data Cambri dge,
  128. (1978). Pychometric theory
  129. (2003). Qua/itative Inquiry in TESOL doi
  130. (1999). Qualitative Evaluation. doi
  131. (2001). Qualitative Market Research; A Comprehensive Guide. Sage Publications,
  132. (1994). Questionnaim Design: a Practical Introduction.
  133. (1977). Reading as Problem Solving: An Investigation of Strategies. doi
  134. (1983). Reconsidering research on learning from media. doi
  135. (1993). Reducing cognitive load in multimedia navigation.
  136. (2001). Reflection. Turning experience into learning,
  137. (1979). Reliability and Validity Assessment, doi
  138. (2005). Research and Evaluation in Education and Prychology: Integrating Diversity withQuantitative, Qualitative, and Mixed Methods. 2"edn. Thousand Oaks,
  139. (2003). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, (2" Ed).
  140. (1992). Research Methods in Language Learning. doi
  141. (1996). Roles for trainers and trainees in Computer-mediated courses.
  142. (1989). Second Language Research Methods.
  143. (1997). Seeking consensus in coursebook evaluation', doi
  144. (1986). Selecting a coursebook: a realistic approach',
  145. (1979). Selecting and evaluating a textbook',
  146. (2005). Sky Pronounciation suite, Sky software house //www. skysoftwarehouse. com/programs. html>
  147. (1978). Sociology in the classroom. In doi
  148. (2004). Soda! Research Methods. (21d Ed)
  149. (1996). Software evaluation: A situated approach. doi
  150. (2002). Speech in Action, doi
  151. (1998). Squaring the circle - reconciling materials as constraint with materials as empowerment, '
  152. (1995). Standards and software-ergonomic evaluation. In
  153. (2003). Stereotype Threat and Race of Interviewer Effects in a Survey on Political Knowledge. doi
  154. (2002). Student perceptions on language learning in a technological environment: implications for the new millennium. ' Language Learning & Technolo (ii /1:
  155. (2003). Students' evaluation of CALL software programs, EducationalMedia Internationa4 40,3 /4,293- 304 Laurillard,
  156. (1993). SU1\II: the Software Usability Measurement Inventory. doi
  157. (1993). Successful Focus Groups: Advancing the State of the Art. doi
  158. (1993). SUMI user handbook.
  159. (1988). Survey review: software for English language teaching. doi
  160. (1991). Talking to questionnaires: pragmatic models in questionnaire design. In
  161. (1956). Taxonomy of Educational Objectives, the Classjcation of Educational Goals - Handbook I: Cognitive Domain. doi
  162. (1996). Teaching as a research-base profession: Possibilities and prospects. Teacher Training Agency Annual Lecture. London: The Teacher Training Agency (ITA).
  163. (1991). Textbook selection and evaluation',
  164. (1998). Textbooks: help or hindrance in teaching?,
  165. (1998). The Analysis of Navigation Patterns doi
  166. The Cognitive Walkthro ugh: Apractitionersguide.
  167. (1967). The countenance of educational evaluations. Teachers College Record,
  168. (1983). The cultural relativity of organizational practices and theories. doi
  169. (1988). The Design, Development, and Evaluation of Instructional Softwarr.
  170. (1991). The effectiveness of research on computer-assisted instruction and computer-assisted language learning. doi
  171. (1997). The empirical evaluation of language teaching materials English Language Teaching doi
  172. (1999). The evaluation of communicative tasks',
  173. (1990). The evaluation of hardware and software, doi
  174. (1990). The focussed interview (2" Ed) doi
  175. (1987). The group depth interview: Principles and practices. Englewood Cliffs NJ:
  176. (1993). The Handbook forFocus Gmup Research.
  177. (1994). The methodology of focus groups: the importance of interaction between research participants', doi
  178. (1998). The price of everything and the value of nothing. trends in language program evaluation. In
  179. (1976). The Pychology of Memory,
  180. (1983). The role of metacognition in reading and studying. In J. Orsany (Ed) Reading comprehension: From research to practice.
  181. (1997). The significance of pauses in written discourse: a comparison of native speaker and learner writing. ' In
  182. (1995). The Software usability measurement inventory- background and usage",
  183. (1997). Towards a Theory of the Effectiveness of Multimedia Systems, doi
  184. (1993). Tracking and analysing learn er-computerinteraction. In
  185. (1990). Understanding Educational Evaluation.
  186. (1988). Understanding Research in Second Language Learning: A teacher''sguide to , statistics and research design. Cambridge.
  187. (2002). Units of productionin writing; evidence of topic `framing' in on line writing research". reading Working Papers in Lsnguistics.
  188. (1993). Usability Engineering, doi
  189. (1994). Usability Inspection Methods doi
  190. (1999). Usability Testing for Developing Effective Interactive Multimedia Software: Concepts, Dimensions, and Procedures'.
  191. (2002). User-centred software evaluation methodologies,
  192. (1985). Using Computers in Language Learning: A Teacher's Guide (2'dedn). London: Centre for information on Language Teaching Research. doi
  193. (2003). Using Focus Groups in Research, Continuum: London,
  194. (1996). Utilizing multimedia toolbook 3.0,
  195. (1983). Verbal reports as data: A focus on retrospection'
  196. (1987). Verbal Reports on 'T'hinking',
  197. (1994). Visualizing Techniques for Examining Learner Interactions with Hypermedia Environments' (ERIC Document Reproduction Service No.
  198. (2000). Visuospatial working memory in learning from multimedia systems. doi
  199. (1994). W(1994) EducationalMultiMedia in a Networked Society
  200. (1998). What do teachers really want from coursebooks?,
  201. (1987). What's underneath?: an interactive view of materials evaluation',
  202. (1987). Which materials? A consumer's and designer's guide',

To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.