Researching for better instructional methods using AB experiments in MOOCs: results and challenges

We conducted two AB experiments (treatment vs. control) in a massive open online course. The first experiment evaluates deliberate practice activities (DPAs) for developing problem solving expertise as measured by traditional physics problems. We find that a more interactive drag-and-drop format of DPA generates quicker learning than a multiple choice format but DPAs do not improve performance on solving traditional physics problems more than normal homework practice. The second experiment shows that a different video shooting setting can improve the fluency of the instructor which in turn improves the engagement of the students although it has no significant impact on the learning outcomes. These two cases demonstrate the potential of MOOC AB experiments as an open-ended research tool but also reveal limitations. We discuss the three most important challenges: wide student distribution, “open-book” nature of assessments, and large quantity and variety of data. We suggest possible methods to cope with those.Google (Firm)Massachusetts Institute of Technolog

Learning Experiments Using AB Testing at Scale

We report the one of the first applications of treatment/control group learning experiments in MOOCs. We have compared the efficacy of deliberate practice-practicing a key procedure repetitively-with traditional practice on "whole problems". Evaluating the learning using traditional whole problems we find that traditional practice outperforms drag and drop, which in turn outperforms multiple choice. In addition, we measured the amount of learning that occurs during a pretest administered in a MOOC environment that transfers to the same question if placed on the posttest. We place a limit on the amount of such transfer, which suggests that this type of learning effect is very weak compared to the learning observed throughout the entire course.Google (Firm)Massachusetts Institute of TechnologyNational Science Foundation (U.S.

Optimal Estimators Of The Position Of A Mass Extinction When Recovery Potential Is Uniform

Numerous methods have been developed to estimate the position of a mass extinction boundary while accounting for the incompleteness of the fossil record. Here we describe the point estimator and confidence interval for the extinction that are optimal under the assumption of uniform preservation and recovery potential, and independence among taxa. First, one should pool the data from all taxa into one combined supersample. Next, one can then apply methods proposed by Strauss and Sadler (1989) for a single taxon. This gives the optimal point estimator in the sense that it has the smallest variance among all possible unbiased estimators. The corresponding confidence interval is optimal in the sense that it has the shortest average width among all possible intervals that are invariant to measurement scale. These optimality properties hold even among methods that have not yet been discovered. Using simulations, we show that the optimal estimators substantially improve upon the performance of other existing methods. Because the assumptions of uniform recovery and independence among taxa are strong ones, it is important to assess to what extent they are satisfied by the data. We demonstrate the use of probability plots for this purpose. Finally, we use simulations to explore the sensitivity of the optimal point estimator and confidence interval to nonuniformity and lack of independence, and we compare their performance under these conditions with existing methods. We find that nonuniformity strongly biases the point estimators for all methods studied, inflates their standard errors, and degrades the coverage probabilities of confidence intervals. Lack of independence has less effect on the accuracy of point estimates as long as recovery potential is uniform, but it, too, inflates the standard errors and degrades confidence interval coverage probabilities

Validating the pre/post-test in a MOOC environment

A standard method for measuring learning is to administer the same assessment before and after instruction. This pre/post-test technique is widely used in education research and has been used in our introductory physics MOOC to measure learning. One potential weakness of this paradigm is that post-test performance gains may result from exposure on the pre-test instead of instruction. This possibility is exacerbated in MOOCs where students receive multiple attempts per item, instant correct/incorrect feedback, and unlimited time (until the due date). To find the size of this problem in our recent MOOCs, we split the student population into two groups, each of which received identical post-tests but different subsets of post-test items on their group pre-test. We report a small overall advantage (2.9% ± 1.7%) on post-test items due to pre-test exposure. However, this advantage is not robust and is strongly diminished when one obviously anomalous item is removed

Data from: Adaptive credible intervals on stratigraphic ranges when recovery potential is unknown

Numerous methods exist for estimating the true stratigraphic range of a fossil taxon based on the stratigraphic positions of its fossil occurrences. Many of these methods require the assumption of uniform fossil recovery potential—that fossils are equally likely to be found at any point within the taxon's true range. This assumption is unrealistic, because factors such as stratigraphic architecture, sampling effort, and the taxon's abundance and geographic range affect recovery potential. Other methods do not make this assumption, but they instead require a priori quantitative knowledge of recovery potential that may be difficult to obtain. We present a new Bayesian method, the Adaptive Beta method, for estimating the true stratigraphic range of a taxon that works for both uniform and non-uniform recovery potential. In contrast to existing methods, we explicitly estimate recovery potential from the positions of the occurrences themselves, so that a priori knowledge of recovery potential is not required. Using simulated datasets, we compare the performance of our method with existing methods. We show that the Adaptive Beta method performs well in that it achieves or nearly achieves nominal coverage probabilities and provides reasonable point estimates of the true extinction in a variety of situations. We demonstrate the method using a dataset of the Cambrian mollusc Anabarella

Version of Fig. 6 accessible to color-deficient readers

Black and white version of Figure 6

Adaptive Credible Intervals On Stratigraphic Ranges When Recovery Potential Is Unknown

Numerous methods exist for estimating the true stratigraphic range of a fossil taxon based on the stratigraphic positions of its fossil occurrences. Many of these methods require the assumption of uniform fossil recovery potential—that fossils are equally likely to be found at any point within the taxon\u27s true range. This assumption is unrealistic, because factors such as stratigraphic architecture, sampling effort, and the taxon\u27s abundance and geographic range affect recovery potential. Other methods do not make this assumption, but they instead require a priori quantitative knowledge of recovery potential that may be difficult to obtain. We present a new Bayesian method, the Adaptive Beta method, for estimating the true stratigraphic range of a taxon that works for both uniform and non-uniform recovery potential. In contrast to existing methods, we explicitly estimate recovery potential from the positions of the occurrences themselves, so that a priori knowledge of recovery potential is not required. Using simulated datasets, we compare the performance of our method with existing methods. We show that the Adaptive Beta method performs well in that it achieves or nearly achieves nominal coverage probabilities and provides reasonable point estimates of the true extinction in a variety of situations. We demonstrate the method using a dataset of the Cambrian mollusc Anabarella

R code

R code for Adaptive Beta Method (v38.0

Version of Fig. 5 accessible to color-deficient readers

Black-and-white version of Figure 5