Location of Repository

Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies

By Barbara Means, Yuki Toyama, Robert Murphy, Marianne Bakia and Karla Jones


A systematic search of the research literature from 1996 through July 2008 identified more than a thousand empirical studies of online learning. Analysts screened these studies to find those that (a) contrasted an online to a face-to-face condition, (b) measured student learning outcomes, (c) used a rigorous research design, and (d) provided adequate information to calculate an effect size. As a result of this screening, 51 independent effects were identified that could be subjected to meta-analysis. The meta-analysis found that, on average, students in online learning conditions performed better than those receiving face-to-face instruction. The difference between student outcomes for online and face-to-face classes—measured as the difference between treatment and control means, divided by the pooled standard deviation—was larger in those studies contrasting conditions that blended elements of online and face-to-face instruction with conditions taught entirely face-to-face. Analysts noted that these blended conditions often included additional learning time and instructional elements not received by students in control conditions. This finding suggests that the positive effects associated with blended learning should not be attributed to the media, per se. An unexpected finding was the small number of rigorous published studies contrasting online and face-to-face learning conditions for K–12 students. In light of this small corpus, caution is required in generalizing to the K–12 population because the results are derived for the most part from studies in other settings (e.g., medical training, higher education)

Topics: T Technology (General), L Education (General)
Publisher: Centre for Learning Technology
Year: 2009
OAI identifier: oai:generic.eprints.org:629/core5

Suggested articles



  1. (1999). 22–24. Are academic behaviors fostered in Web-based environments?
  2. (2006). A comparative study of three instructional modalities in a computer programming course: Traditional instruction, Web-based instruction, and online instruction. PhD diss.,
  3. (2007). A comparison of anonymous versus identifiable e-peer review on college student writing performance and the extent of critical feedback.
  4. (2000). A comparison of collaborative learning and audience awareness in two computer-mediated writing environments.
  5. (2007). A comparison of face-to-face and Webbased classrooms.
  6. (2006). A comparison of learning outcomes by ‘in-course’ evaluation techniques for an online course in a controlled environment.
  7. (2002). A comparison of traditional and World Wide Web methodologies, computer anxiety, and higher order thinking skills in the inservice training of Mississippi 4-H extension agents. doi
  8. (2000). A meta-analysis of the effectiveness of telecourses in distance education. doi
  9. (1992). A power primer. doi
  10. (2007). A study of the effectiveness of the Louisiana Algebra I online course. doi
  11. (2008). A study on learning effect among different learning styles in a Web-based lab of science for elementary school students. doi
  12. (2000). Achievement predictors for a computer-applications module delivered online.
  13. (2007). Acquisition of information online: Knowledge, navigation and learning outcomes.
  14. (1994). Administrative barriers to adoption of distance education. doi
  15. (2008). An experimental study of simulated Web-based threats and their impact on knowledge communication effectiveness. doi
  16. (2006). An interactive Web-based curriculum on evidence-based medicine: Design and effectiveness. Family Medicine.
  17. (2006). An overview of current findings from empirical research on online teacher professional development.
  18. (2007). Applying Web-enabled problem-based learning and self-regulated learning to enhance computing skills of Taiwan’s vocational students: A quasi-experimental study of a short-term module.
  19. (2004). Are computer-assisted teaching methods effective? doi
  20. (2007). Assessing the success rate of students using a learning management system together with a collaborative tool in Web-based teaching of programming languages. doi
  21. (1999). Asynchronous computer-mediated communication versus face-to-face collaboration: Results on student learning, quality and satisfaction.
  22. (1998). Audiographic telecourses for the Web: An experiment. doi
  23. (2004). Can teachers learn through enquiry online? Studying professional development in science delivered online and on-campus. doi
  24. (2002). Classroom action research: A case study assessing students’ perceptions and learning outcomes of classroom teaching versus online teaching.
  25. (2004). Community of practice: A metaphor for online design. In Designing for virtual communities in the service of learning,
  26. (2007). Comparing and combining real and virtual experimentation: An effort to enhance students’ conceptual understanding of electric circuits. doi
  27. (2007). Comparing strategies for teaching abstract concepts in an online tutorial. doi
  28. (2002). Comparison and evaluation of online and classroom instruction in elementary science teaching methods courses. Paper presented at the 1st Northwest NOVA Cyber-Conference,
  29. (2007). Comparison of fixed-item and response-sensitive versions of an online tutorial.
  30. (2006). Comparison of Web-based instructional design strategies in a pain management program for nursing professional development.
  31. (2006). Comprehensive Meta-Analysis (Version 2.2.027). [Computer software].
  32. (2001). Computer conferencing: An environment for collaborative project-based learning in distance education. doi
  33. (2007). Construction of a learning environment supporting learners’ reflection: A case of information seeking on the Web. doi
  34. controlled trial of spaced education to urology residents in the United States and Canada. doi
  35. (1999). Design of support tools and knowledge building in a virtual university course: Effect of reflection and self-explanation prompts.
  36. (2008). Developing and evaluating an interactive multimedia instructional tool: Learning outcomes and user experiences of optometry students.
  37. (1999). Developing online courses: A comparison of Web-based instruction with traditional instruction.
  38. (2001). Digital skill training research: Preliminary guidelines for distributed learning (Final report).
  39. (2008). Discussing patient management online: The impact of roles on knowledge construction for students interning at the paediatric ward. doi
  40. (2003). Distance education and career and technical education: A review of the research literature. doi
  41. (2005). Distance education courses for public elementary and secondary school students:
  42. (2001). Do no harm: A comparison of the effects of online vs. traditional delivery media on a science course.
  43. (2006). Does it improve high school students’ ability to perform cardiopulmonary resuscitation in a simulated environment? doi
  44. (2005). Does it work?”: The impact of technology and professional development on student achievement.
  45. (2007). Dynamic language learning: Comparing mobile language learning with online language learning. doi
  46. (2007). E-learning: A comparative study for knowledge apprehension among nurses. Revista Latino-Americana de Enfermagem 15:397–403. doi
  47. (2007). ED PACE final report. Submitted to the West Virginia Department of Education. San Francisco: Author. www.rockman.com/projects/146.ies.edpace/finalreport (accessed
  48. (2007). Educating generalist physicians about chronic pain: Live experts and online education can provide durable benefits. doi
  49. (2008). Effect of embedded streaming video strategy in an online learning environment on the learning of neuroscience.
  50. (2008). Effectiveness of a blended e-learning cooperative approach in an Egyptian teacher education programme. doi
  51. (2008). Effectiveness of e-learning in continuing medical education for occupational physicians. doi
  52. (2006). Effects of a distance learning program on physicians’ opioid- and benzodiazepine-prescribing skills. doi
  53. (2005). Effects of a Web-based teaching method on undergraduate nursing students’ learning of electrocardiography.
  54. (2007). Effects of advance organizers on learning and retention from a fully Web-based class. PhD diss.,
  55. (2007). Effects of online versus face-to-face professional development with a teambased learning community approach on teachers’ application of a new instructional practice.
  56. (2004). Effects of peer-controlled or externally structured and moderated online collaboration on group problem solving processes and related individual attitudes in wellstructured and ill-structured small group problem solving in a hybrid course. PhD diss.,
  57. (2006). EMentoring for Student Success: Online mentoring and professional development for new science teachers.
  58. (2007). Employees’ attitudes toward e-learning: Implications for policy in industry environments.
  59. (2006). Enhancement of secondary special education teachers’ knowledge and competencies in working with families through online training modules. PhD diss.,
  60. (2008). Enhancing online collaborative argumentation through question elaboration and goal instructions. doi
  61. (2008). Enhancing self-perceived effects using Web-based portfolio assessment. doi
  62. (2007). Enhancing Web-based language learning through self-monitoring. doi
  63. (2005). Epistemic and social scripts in computer supported collaborative learning. doi
  64. (1999). Evaluating the serial migration of an existing required course to the World Wide Web. doi
  65. (2005). Evaluating Web-supported learning versus lecture-based teaching: Quantitative and qualitative perspectives. doi
  66. (1997). Evaluation by electronic mail. Paper presented at the annual meeting of the Mid-South Educational Research Association,
  67. (2008). Evaluation of a particle repositioning maneuver web-based teaching module. doi
  68. (2007). Evaluation of a Web-based intervention for improving HIV/AIDS knowledge in rural Yunnan, doi
  69. (2003). Evaluation of an interactive tutorial for teaching hypothesis testing concepts. doi
  70. (2001). Evaluation of distance-learning environments: Impact of usability on student performance.
  71. (2001). Exploring the academic self within an electronic mail environment.
  72. (2007). Exploring the use of individualized, reflective guidance in an educational multi-user virtual environment. doi
  73. (2007). Final report: A comprehensive assessment of Florida virtual school. Tallahassee, Fla.: Florida TaxWatch Center for Educational Performance and Accountability.
  74. (2004). How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. doi
  75. (1999). How people learn: Brain, mind, experience, and school.
  76. (2000). Implicit and explicit dialogue structuring in virtual learning groups. doi
  77. (2007). Instructional methods and cognitive and learning styles in Web-based learning: Report of two randomised trials. doi
  78. (2006). Instructional video in elearning: Assessing the impact of interactive video on learning effectiveness. doi
  79. (2005). Interactive multimedia-based e-learning: A study of effectiveness.
  80. (2008). Interactive spaced education versus Web-based modules for teaching urology to medical students: a randomized controlled trial. doi
  81. (2007). K–12 online learning: A survey of U.S. school district administrators. Boston: Sloan Consortium. http://www.sloan-c.org/publications/survey/K-12_06.asp (accessed
  82. (2007). K–12 online learning: A survey of U.S. school district administrators. Boston: Sloan Consortium. http://www.sloan-c.org/publications/survey/K12_06.asp (accessed
  83. (2007). Laboratories of reform: Virtual high schools and innovation in public education.
  84. (2007). Learner-instructor telephone interaction: Effects on satisfaction and achievement of online students. doi
  85. (1998). Learning in cyberspace: Shaping the future.
  86. (2006). Learning outcomes in medical informatics: Comparison of a WebCT course with ordinary web site learning material. doi
  87. (2007). Learning stoichiometry: A comparison of text and multimedia instructional formats. PhD diss.,
  88. (2006). Learning styles and formative assessment strategy: Enhancing student achievement in Web-based learning. doi
  89. (2007). Liberating the spacing effect from the laboratory: A practical application in a worldwide Web-based religious education volunteer-teacher training program. PhD diss.,
  90. (2002). Library instruction and graduate professional development: Exploring the effect of learning environments on self-efficacy and learning outcomes.
  91. (2006). Long-term impact of four different strategies for delivering an online curriculum about herbs and other dietary supplements.
  92. (1994). Media will never influence learning.
  93. (2002). Multimedia comprehension skill predicts differential outcomes of Web-based and lecture courses. doi
  94. (2006). Multimedia-enhanced instruction in online learning environments.
  95. (2004). Online communities: Common ground and critical differences in designing technical environments. In Designing for virtual communities in the service of learning, doi
  96. (2004). Online compared to face-to-face teacher preparation for learning standards-based planning skills. doi
  97. (2006). Online professional development for teachers: Emerging models and methods. doi
  98. (2003). Online training: An evaluation of the effectiveness and efficiency of training law enforcement personnel over the Internet.
  99. (2008). Online vs. face-to-face discussion in a Web-based research methods course for postgraduate nursing students: A quasi-experimental study. doi
  100. (2005). Online vs. live methods for teaching difficult airway management to anesthesiology residents. doi
  101. (2008). Online weight training. doi
  102. (2008). Pedagogical utilization and assessment of the statistic online computational resource in introductory probability and statistics courses.
  103. (2000). Practical meta-analysis.
  104. (2008). Preparing child care health consultants to address childhood overweight: A randomized controlled trial comparing Web to in-person training. doi
  105. (2008). Pressure ulcers: E-learning to improve classification by nurses and nursing students. doi
  106. (2003). Qualitative analysis of the overhand throw by undergraduates in education using a distance learning computer program. doi
  107. (2006). Randomized controlled trial comparing four strategies for delivering e-curriculum to health care professions.
  108. (2008). Randomized controlled trial of an Internet-based versus face-to-face dyspnea self-management program for patients with chronic obstructive pulmonary disease: Pilot study. doi
  109. (1983). Reconsidering research on learning from media.
  110. (2005). Replacing lecture with Web-based course materials. doi
  111. (2005). Scaffolding peer-questioning strategies to facilitate metacognition during online small group discussion. doi
  112. (2007). Scaffolding the writing of students with disabilities through procedural facilitation: Using an Internet-based technology to improve performance. doi
  113. (1985). Statistical methods for meta-analysis.
  114. (2001). Student achievement and performance levels in online education research study.
  115. (2004). Teacher professional development: Issues and trends.
  116. (2006). Teaching courses online: A review of research. doi
  117. (2004). Teaching in cyberspace: Online versus traditional instruction using a waiting-list experimental design. doi
  118. (2008). Teaching science as a language: A “content-first” approach to science teaching.
  119. (2007). Teaching veterinary radiography by e-learning versus structured tutorial: A randomized, single-blinded controlled trial. doi
  120. (2007). Technical details of WWC-conducted computations.
  121. (2007). Technology-assisted learning and learning style: A longitudinal field experiment. Systems, man and cybernetics, part A. doi
  122. (2008). Technology-based distance education courses for public elementary and secondary school students:
  123. (2007). The accelerate study: The longitudinal effect of speed of processing training on cognitive performance of older adults. doi
  124. (2006). The comparative effectiveness of Web-based and classroom instruction: A meta-analysis. Personnel Psychology 59:623–64. doi
  125. (2007). The dynamics involved in Web-based learning environment (WLE) interface design and human-computer interactions (HCI): Connections with learning performance.
  126. (2007). The effect of an electronic performance support system and training as performance interventions. PhD diss.,
  127. (2006). The effect of cognitive styles upon the completion of a visually-oriented component of online instruction. PhD diss.,
  128. (2005). The effect of three computer conferencing designs on critical thinking skills of nursing students. doi
  129. (2006). The effect of using guided questions and collaborative groups for complex problem solving on performance and attitude in a Web-enhanced learning environment. PhD diss.,
  130. (2007). The effectiveness and user perception of 3-dimensional digital human anatomy in an online undergraduate anatomy laboratory.
  131. (2004). The effectiveness of an interactive multimedia learning tool on nursing students’ math knowledge and self-efficacy. doi
  132. (2002). The effectiveness of discussion forums in online learning.
  133. (2001). The effectiveness of interactive distance education technologies in K–12 learning: A meta-analysis.
  134. (2004). The effectiveness of Web assisted English language instruction on the achievement and attitude of the students.
  135. (2003). The effectiveness of Web-based training.
  136. (2007). The effects of concept mapping as advance organizers in instructional designs for distance learning programs. PhD diss.,
  137. (2003). The effects of different levels of interaction on the achievement and motivational perceptions of college students in a Web-based learning environment.
  138. (2004). The effects of distance education on K–12 student outcomes: A meta-analysis. Naperville, Ill.: Learning Point Associates. http://www.ncrel.org/tech/distance/index.html (accessed
  139. (2007). The effects of instructor immediacy in online learning environments. PhD diss.,
  140. (2008). The effects of scaffolding student’s problem-solving process via question prompts on problem solving and intrinsic motivation in an online learning environment.
  141. (2005). The effects of technology on reading performance in the middle school grades: A meta-analysis with recommendations for policy. Naperville, Ill.: Learning Point Associates. (ERIC Document Reproduction Service No.
  142. (2006). The effects of using computer-based distance education for supplemental instruction compared to traditional tutorial sessions to enhance learning for students at-risk for academic difficulties. PhD diss.,
  143. (2004). The effects of Web-based learning on struggling EFL college writers. doi
  144. (2007). The effects of Web-based social networks on student achievement and perception of collaboration at the middle school level. PhD diss.,
  145. (1998). The effects of World Wide Web instruction and traditional instruction and learning styles on achievement and changes in student attitudes in a technical writing in agricommunication course. doi
  146. (2007). The impact of a computer simulated case study on nurse practitioner students’ declarative knowledge and clinical performance. PhD diss.,
  147. (2008). The impact of computer-based simulation within an instructional sequence on learner performance in a Web-based environment. PhD diss.,
  148. (2007). The impact of Web-based worked examples and selfexplanation on performance, problem solving, and self-efficacy. doi
  149. (2007). Traditional and virtual microscopy compared experimentally in a classroom setting. doi
  150. (1999). Two Canadian approaches to teaching biology, chemistry, mathematics and physics to senior high school students in virtual classes.
  151. (2003). Using technology to enhance a course: The importance of interaction.
  152. (2004). Using technology to promote self-efficacy for healthy eating in adolescents. doi
  153. (2007). Web-based education in systems-based practice: A randomized trial.
  154. (2005). Web-based learning in residents’ continuity clinics: A randomized, controlled trial. doi
  155. (2006). Web-based learning versus standardized patients for teaching clinical diagnosis: A randomized, controlled, crossover trial. Teaching and Learning in doi
  156. (2003). Wellness NutriFit online learning in physical education for high school students.
  157. (2005). What makes the difference? A practical analysis of research on the effectiveness of distance education. doi

To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.