What Makes the Difference? A Practical Analysis of Research on the Effectiveness of Distance Education

This article reports findings of a meta-analytical study of research on distance education. The purpose of this study was to identify factors that affect the effectiveness of distance education. The results show that although the aggregated data of available studies show no significant difference in outcomes between distance education and face-to-face education as previous research reviews suggest, there is remarkable difference across the studies. Further examination of the difference reveals that distance education programs, just like traditional education programs, vary a great deal in their outcomes, and the outcome of distance education is associated with a number of pedagogical and technological factors. This study led to some important data-driven suggestions for and about distance education. History is a mirror of the past and a lesson for the present. -Persian proverb 2004; Lockee, Although the call for a new paradigm of research in distance education has not stopped researchers from conducting comparison studies in distance education, as publications of this nature continue to appear in scholarly journals and professional conferences Traditionally, the lens used to synthesize media comparison studies has often in essence been the same as the one used by comparison studies: the difference finder. A number of synthesis studies have been conducted in the field of distance education These studies have undoubtedly helped to promote distance instruction as a viable form of education with the same quality as its face-to-face counterpart; they have also led to the belief that previous distance education research is of low quality and has little to offer in terms of practical guidance for improving practice. Consequently, it is suggested that we should just discard it and move forward. However, upon closer examination, these studies reveal that individual studies indeed found significant difference between distance and face-to-face instruction. It is likely that a systematic analysis of what may account for the different findings across studies could provide us with practical guidance for improving practice. The no-significant-difference conclusion was primarily drawn from two types of analyses: summary of studies that found no significant difference What Makes the Difference? 1839 A more valid and reliable way to synthesize the literature is meta-analysis The considerable heterogeneity in the studies clearly indicates that there is indeed significant difference in learning outcomes of distance and faceto-face education. Individually many studies found significant differences between distance and face-to-face education, some favoring distance education and others face-to-face education. In fact, contrary to Russell' § claim, it is rarely the case that the individual studies included in the metaanalyses conducted by The significant heterogeneity of achievement begs the question: Why did some studies find distance education students had better achievement than their counterparts in traditional classrooms and some find the opposite? To answer this question, we need to further examine the characteristics of each individual study. We know that distance education programs vary a great deal in content, learner characteristics, instructor characteristics, and delivery method. By examining these variables and the degree to which they influence learning outcomes, we may be able to arrive at what distance education research is encouraged to do: find useful guidance for practice and research. Machtmes and Asher's study suggests a promising way to make use of the distance education literature, but it has a number of significant limitations. First, it has an extremely small sample. With only 19 studies from 13 publications that span 30 years (from 1963 to 1993), the power of the study is extremely limited. Second, the study is limited to video-based/televised distance programs, whereas in recent years distance education has employed many other technologies, including computer conferencing, the World Wide Web, and CD-ROMs. "In order to identify which features impact student learning," they suggest, "researchers need to systematically identify and evaluate the technological and instructional features of all delivery systems" (p. 42). Third, the coding of the variables in the study is all categorical, whereas we know that some of the features vary on a continuum. For example, the availability of the instructor is not a simple yes or no, because some distance education programs may have instructors available more frequently than others. Thus treating it as a continuous variable more accurately reflects the reality. The last but perhaps most significant limitation is the lack of a well-developed framework for identifying possible features that may contribute to learner achievement. The purpose of the present study is in some way a continuation of Machtmes and Asher's, but with more emphasis on systematically examining how different features of previous studies of distance education affect learning outcomes so as to inform future practice and research. To avoid the limitations of previous synthesis studies, we examined a much larger body of research, applied more sophisticated statistical procedures, and developed a more systematic analytical framework. In the remainder of this article, we describe the methods, findings, and implications of the present study. What Makes tde Difference? 1841 METHODOLOGY LITERATURE SEARCH AND SELECTION Studies included in the research synthesis were identified through a threestep process. First, we conducted a thorough search for all studies included in the Education Resources Information Center through FirstSearch with the following keywords: distan* and education, distan* and learning, distan* and teaching, distan* and instruction, online and education, online and learning, online and teaching, online and instruction, on-line and education, on-line and learning, on-line and teaching, on-line and instruction, web-based and education, web-based and learning, webbased and teaching, web-based and instruction, virtual and education, virtual and learning, virtual and teaching, virtual and instruction. The search identified 8,840 potentially relevant articles. Citation information for all 8,840 articles was then transferred into EndNote (version 5.0; ISI ResearchSoft, 2001) to build the first database. At the second stage, the database was further examined based on the following criteria: 1. The article had to be published in a journal. The decision to include only journal articles was based on the concern of study quality. Previous research reviews on distance education had pointed out the low-quality problem of most studies. We believed that journal articles were of higher quality because of peer review procedures. Only including journal articles may result in publication bias, but we believed that the risk was minimal, as there had not been a dominant paradigm for distance education over the years to cause a certain bias against or for positive, negative, or nonsignificant findings. 2. The article must have had complete reference information (author, date, source, etc.). 3. The article had to include at least one evaluation study of distance education. The specific outcome measured was not limited. 4. Thel article must have had at least one comparison study on distance education and face-to-face education. Studies in which students' own pretreatment scores served as controls for their posttreatment scores and those in which one distance course was compared with another distance course were excluded. 5. The article must have had some empirical data about the learning outcomes. Articles were not included if they merely describe a distance education course. 1842 Teachers College Record 6. The article had to include enough statistical information for computing an effect size. The specific information we were looking for was mean, standard deviation, and sample size for both the distance education group and the face-to-face group, or t value, F value and degree of freedom (df). A total of 1,100 articles that either were not journal articles or didn't have complete reference information were removed from the database. Then the research team read the abstracts of the remaining 7,740 articles, 6,365 of which were removed because they didn't meet criterion 3 or 4. Articles on which a clear decision could not be reached at this stage were kept in the database to be dealt with at a later stage. A total of 1,375 references were left after this selection. At the third stage, the research team collected and read the 1,375 articles and excluded those that didn't have empirical data. As a result, 421 articles were left after this elimination. The research team then read and coded these 421 articles and found that only 49 articles contained sufficient information for calculating the effect size. For fear of missing some articles that might actually have had the information because of the large number of articles and complexity of the database, the research team examined the 421 articles once more and identified 2 more articles that had complete information for calculating the effect size. Ancestry search was conducted, but no extra articles that met all the criteria were found. Thus, 51 journal articles were included for the analysis. ANALYTICAL FRAMEWORK To identify those methodological and substantive characteristics that may be responsible for significant variations in the findings, a detailed analytical framework was developed through an iterative process. The framework was developed based on our understanding of possible sources of variation in the studies. Heterogeneity in outcomes across studies can come from three sources: the publication, the study, and the instruction. For publication and study features, we started with Stock's (1994) seven categories for describing research reports: report identification, the setting of the study, participants, methodology, treatment characteristics, statistical outcomes or effect sizes, and coding process. Instructional features refer to the characteristics of the distance education program under study. It has been argued that distance education should be considered as education at a distance (Shale, 1990): In sum, distance education ought to be regarded as education at a distance. All of what constitutes the process of education when teacher In other words, the quality of distance education programs is influenced by the same set of factors that affect the quality of face-to-face education. Schwab (1983) characterizes education in terms of four common places of education: teacher, student, what is taught, and milieux of teaching-learning. This characterization is also applicable to the study of distance education. Whereas teacher, student, and what is taught remain pretty much the same as in face-to-face education, the milieux of teaching-learning are different between distance and face-to-face education in that the milieux of teaching-learning of distance education are mostly mediated through some kind of technology. Hence we describe the milieux of teaching-learning in terms of the format and method of delivery. Finally, we used the grounded theory Evidence of Effectiveness There are different ways to measure the effectiveness of distance education programs. Studies in distance education thus differ in what they used as evidence of effectiveness and the reliability and validity of the evidence used. The variation in what was measured and the quality of the measurement may explain the heterogeneity of outcomes. Outcome measures. Information about what has been used to assess the effectiveness of distance programs was collected for each study. A study could use one or more of the following measures: grades, quizzes, independent/standardized tests, student satisfaction, instructor satisfaction, dropout rate, student evaluation of learning, student evaluation of course, and external evaluation. Grades usually are the final scores students received for the class. Student evaluation of learning is students' perception of how much they learned from the course, which can be significantly different from the grades they received. Source of instrument. The source of instruments used to measure effectiveness can affect the final outcomes in that instruments from different sources may have different levels of reliability and validity. We identified four sources of the most frequently used measures: commercial testing agencies, the researcher (the author of the article), the instructor of the course, and publishers of textbooks that include assessment items. Study design. The design of a study is a good indication of its quality and thus the quality of the results. We were curious about whether a certain type of design is associated with the study results; thus we coded the studies into two categories: true experimental or quasi-experimental. The differentiating characteristic between the two designs is whether random sampling method was used. Study results. To calculate effect sizes, the results of each study were recorded in the database. The study results include means, standard deviations, t values, F values, and r values, depending on what is reported in the primary study. Factors Affecting Effectiveness Factors that may affect the effectiveness of a distance education program can be categorized into two groups: publication features and instructional features. Two publication features were identified as possible factors affecting effectiveness: publication year and instructor as author. Publication year. Previous research found that the time when a study was conducted had a significant correlation with the reported effectiveness Instructor as author. All studies are based on advocacy As mentioned before we used Schwab's four common places of education to guide the identification of instructional features that could potentially affect the effectiveness of distance education programs: the teacher, the student, the curriculum, and the milieux. In each of the four common places were a number of potential factors that could contribute to the outcomes of learning. Teacher: Instructor involvement. The extent to which the instructor of a distance education course is involved in the actual delivery of the content and available for interactions with students during and outside the class sessions is termed "instructor involvement." The level of instructor involvement is perhaps one of the most defining differences between traditional face-to-face education and distance education. In face-to-face education, the instructor generally delivers the content live and interacts with students both in and outside class meetings, whereas in distance education programs the level of instructor involvement varies a great deal, from one extreme where the content is preprogrammed and delivered through some technology means without the actual involvement of an instructor to another where the instructor actually delivers the content live and is available for interactions with students in very much the same fashion as face-to-face education. Interactions between the teacher and students have been found to affect the quality of student experiences and learning outcomes in distance education (Institute for Higher Education Policy, 2000). How content is delivered should also have an effect on student learning experiences and outcomes. This is also a hot topic for distance education programs because one of the appeals of distance education is the potential to increase efficiency by reducing the demand of actual involvement of faculty. Having one faculty member teaching thousands of students (or even more) with the help of broadcasting, recording, and computing technologies has been a dream for many advocates of distance education. However, if a higher level of instructor involvement becomes a requirement for effective programs, the efficiency dream may never be realized. On the other hand, if the level of instructor involvement is found to be irrelevant to student outcomes, it would be unnecessary to assign an instructor to only a small group of students or have him or her be actually involved in the teaching. A preprogrammed video or computer program can accomplish as much. To assess the value of instructor involvement, we hence included this factor. We coded instructor involvement on a scale from 1 (no human involvement; e.g., computer-based training) to 10 (full involvement of a human instructor; e.g., two-way interactive TV courses). What Makes the Difference? 1847 Teacher: Status of the instructor. Another way distance education programs have sought to increase efficiency is to employ nonregular faculty, who normally cost less than regular faculty. Thus we were interested in whether instructor status influenced student learning in distance education programs. Teacher: Training for teaching distance courses. It has been argued that instructors of distance education programs should be trained first, because distance education is a different teaching environment from face-to-face classrooms. Again, the training has cost implications for programs. To test whether training affects student learning, we collected information about teacher training from each study, if that information was available. Student: Education level. We collected information about students' educational attainment level before attending the distance course to examine whether, and if so at which level, certain types of students were more prepared to take distance education courses. What is being taught: Content area. Some content may be more suited for distance education, whereas other content may be better taught in a face-toface course. Interested in whether this assumption was true and if so, what content area was better suited for distance education, we collected information about the content area of each study. A course could be categorized as teaching one of the following subject areas: social science, mathematics, science, medical science, literacy, humanities, business, law, engineering, computer science, teacher education, and skills. (Skills here represented any professional training that didn't fall into other categories.) We coded medical science, business education, and teacher education separately because they had been among the most commonly taught content areas in distance education. The milieux: Instructional level. Distance education programs have been traditionally intended for adults, but recently distance education has expanded to include younger audiences. As a related factor to student characteristics and content, the instructional level of distance education may be associated with its effectiveness. We grouped the distance education programs in each study into nine levels: Grades K-2 (lower elementary), Grades 3-5 (upper elementary), Grades 6-9 (middle school), Grades 10-12 (high school), associate's degree (community college), undergraduate level (4-year college), graduate level, professional development, and military training. We also collected data about whether the course was for credit or not and whether it was for a degree-granting program or not. Teachers College Record The milieux: Interaction type. Interaction type characterizes how instructors and students interact in the distance learning process. There are four types of interaction: asynchronous, in which a time lag exists between the interactions of the instructor and students in that students may ask a question via e-mail, to which the instructor may respond, for example, 2 days later; synchronous, where the potential exists for instructors and students to interact at the same time; noninteractive, where there is no interaction between instructors and students at all; and both synchronous and asynchronous, where the instructor can interact with students both synchronously and asynchronously. The milieux: Media involvement. Distance education programs also vary in the level of technology used. Some programs employ a mixed model, in which part of the instruction is conducted face-to-face whereas some others are delivered via technology. Proponents of the mixed model suggest that some face-to-face contact is necessary or desirable to maintain student motivation and thus a higher quality of education. We were interested in testing this hypothesis. Thus we coded each study's level of media involvement, which was defined as the extent to which a certain instructional delivery system has been mediated by technologies, that is, how frequently technology is used in a program. Media involvement is coded on a scale from 1 (no technology was used) to 10 (instruction was delivered completely with technology). DATA CODING PROCESS AND INTERRATER RELIABILITY Information from complete articles selected for inclusion was coded by the two researchers who were most involved in the development of the framework and rubric. One coded 25 articles and the other coded 26 articles independently. When there was any uncertainty, a third researcher was involved, and an agreement would be reached through discussion. After both researchers completed the coding, 10 articles were randomly selected to test interrater reliability. Both coders coded these 10 articles, and they reached an agreement of 98.3%. Disagreements were solved through discussion. DATA INTEGRATION Effect Size Computation Effect size is a measure of standardized mean difference between two groups. In this study, effect size was computed to estimate the extent of the difference between online learning and face-to-face l

Tests of relativity using a microwave resonator

The frequencies of a cryogenic sapphire oscillator and a hydrogen maser are compared to set new constraints on a possible violation of Lorentz invariance. We determine the variation of the oscillator frequency as a function of its orientation (Michelson-Morley test) and of its velocity (Kennedy-Thorndike test) with respect to a preferred frame candidate. We constrain the corresponding parameters of the Mansouri and Sexl test theory to $\delta - \beta + 1/2 = (1.5\pm 4.2) \times 10^{-9}$ and $\beta - \alpha - 1 = (-3.1\pm 6.9) \times 10^{-7}$ which is equivalent to the best previous result for the former and represents a 30 fold improvement for the latter.Comment: 8 pages, 2 figures, submitted to Physical Review Letters (October 3, 2002

Modelling national HIV/AIDS epidemics: revised approach in the UNAIDS Estimation and Projection Package 2011

Objective: United Nations Programme on HIV/AIDS reports regularly on estimated levels and trends in HIV/AIDS epidemics, which are evaluated using an epidemiological model within the Estimation and Projection Package (EPP). The relatively simple four-parameter model of HIV incidence used in EPP through the previous round of estimates has encountered challenges when attempting to fit certain data series on prevalence over time, particularly in settings with long running epidemics where prevalence has increased recently. To address this, the most recent version of the modelling package (EPP 2011) includes a more flexible epidemiological model that allows HIV infection risk to vary over time. This paper describes the technical details of this flexible approach to modelling HIV transmission dynamics within EPP 2011. Methodology For the flexible modelling approach, the force of infection parameter, r, is allowed to vary over time through a random walk formulation, and an informative prior distribution is used to improve short-term projections beyond the last year of data. Model parameters are estimated using a Bayesian estimation approach in which models are fit to HIV seroprevalence data from surveillance sites. Results: This flexible model can yield better estimates of HIV prevalence over time in situations where the classic EPP model has difficulties, such as in Uganda, where prevalence is no longer falling. Based on formal out-of-sample projection tests, the flexible modelling approach also improves predictions and CIs for extrapolations beyond the last observed data point. Conclusions: We recommend use of a flexible modelling approach where data are sufficient (eg, where at least 5 years of observations are available), and particularly where an epidemic is beyond its peak

Metastability in spin polarised Fermi gases and quasiparticle decays

We investigate the metastability associated with the first order transition from normal to superfluid phases in the phase diagram of two-component polarised Fermi gases.We begin by detailing the dominant decay processes of single quasiparticles.Having determined the momentum thresholds of each process and calculated their rates, we apply this understanding to a Fermi sea of polarons by linking its metastability to the stability of individual polarons, and predicting a region of metastability for the normal partially polarised phase. In the limit of a single impurity, this region extends from the interaction strength at which a polarised phase of molecules becomes the groundstate, to the one at which the single quasiparticle groundstate changes character from polaronic to molecular. Our argument in terms of a Fermi sea of polarons naturally suggests their use as an experimental probe. We propose experiments to observe the threshold of the predicted region of metastability, the interaction strength at which the quasiparticle groundstate changes character, and the decay rate of polarons

Quantum interference of ultrastable twin optical beams

We report the first measurement of the quantum phase-difference noise of an ultrastable nondegenerate optical parametric oscillator that emits twin beams classically phase-locked at exact frequency degeneracy. The measurement illustrates the property of a lossless balanced beam-splitter to convert number-difference squeezing into phase-difference squeezing and, thus, provides indirect evidence for Heisenberg-limited interferometry using twin beams. This experiment is a generalization of the Hong-Ou-Mandel interference effect for continuous variables and constitutes a milestone towards continuous-variable entanglement of bright, ultrastable nondegenerate beams.Comment: 4 pages, 4 figs, accepted by Phys. Rev. Let

Exclusion Statistics in a two-dimensional trapped Bose gas

We briefly explain the notion of exclusion statistics and in particular discuss the concept of an ideal exclusion statistics gas. We then review a recent work where it is demonstrated that a {\em two-dimensional} Bose gas with repulsive delta function interactions obeys ideal exclusion statistics, with a fractional parameter related to the interaction strength.Comment: 10 pages, RevTeX. Proceedings of the Salerno workshop "Theory of Quantum Gases and Quantum Coherence", to appear in a special issue of J.Phys. B, Dec. 200

Runaway dilaton and equivalence principle violations

In a recently proposed scenario, where the dilaton decouples while cosmologically attracted towards infinite bare string coupling, its residual interactions can be related to the amplitude of density fluctuations generated during inflation, and are large enough to be detectable through a modest improvement on present tests of free-fall universality. Provided it has significant couplings to either dark matter or dark energy, a runaway dilaton can also induce time-variations of the natural "constants" within the reach of near-future experiments.Comment: 4 pages, minor change

Observation of anomalous spin-state segregation in a trapped ultra-cold vapor

We observe counter-intuitive spin segregation in an inhomogeneous sample of ultra-cold, non-condensed Rubidium atoms in a magnetic trap. We use spatially selective microwave spectroscopy to verify a model that accounts for the differential forces on two internal spin states. In any simple understanding of the cloud dynamics, the forces are far too small to account for the dramatic transient spin polarizations observed. The underlying mechanism remains to be elucidated.Comment: 5 pages, 3 figure