Beyond A/B Testing: Sequential Randomization for Developing Interventions in Scaled Digital Learning Environments

Randomized experiments ensure robust causal inference that are critical to effective learning analytics research and practice. However, traditional randomized experiments, like A/B tests, are limiting in large scale digital learning environments. While traditional experiments can accurately compare two treatment options, they are less able to inform how to adapt interventions to continually meet learners' diverse needs. In this work, we introduce a trial design for developing adaptive interventions in scaled digital learning environments -- the sequential randomized trial (SRT). With the goal of improving learner experience and developing interventions that benefit all learners at all times, SRTs inform how to sequence, time, and personalize interventions. In this paper, we provide an overview of SRTs, and we illustrate the advantages they hold compared to traditional experiments. We describe a novel SRT run in a large scale data science MOOC. The trial results contextualize how learner engagement can be addressed through inclusive culturally targeted reminder emails. We also provide practical advice for researchers who aim to run their own SRTs to develop adaptive interventions in scaled digital learning environments

Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies

A systematic search of the research literature from 1996 through July 2008 identified more than a thousand empirical studies of online learning. Analysts screened these studies to find those that (a) contrasted an online to a face-to-face condition, (b) measured student learning outcomes, (c) used a rigorous research design, and (d) provided adequate information to calculate an effect size. As a result of this screening, 51 independent effects were identified that could be subjected to meta-analysis. The meta-analysis found that, on average, students in online learning conditions performed better than those receiving face-to-face instruction. The difference between student outcomes for online and face-to-face classes—measured as the difference between treatment and control means, divided by the pooled standard deviation—was larger in those studies contrasting conditions that blended elements of online and face-to-face instruction with conditions taught entirely face-to-face. Analysts noted that these blended conditions often included additional learning time and instructional elements not received by students in control conditions. This finding suggests that the positive effects associated with blended learning should not be attributed to the media, per se. An unexpected finding was the small number of rigorous published studies contrasting online and face-to-face learning conditions for K–12 students. In light of this small corpus, caution is required in generalizing to the K–12 population because the results are derived for the most part from studies in other settings (e.g., medical training, higher education)

The development and testing of a child-inspired advertising disclosure to alert children to digital and embedded advertising

Via three studies, this article aims to develop and test an advertising disclosure which is understandable for children (ages six to 12 years old) and which can alert them to different types of advertising in multiple media formats. First, cocreation workshops with 24 children (ages eight to 11 years old) were held to determine a selection of disclosure designs based on insights from the target group. Second, two eye-tracking studies among 32 children (ages six to 12 years old) were conducted to test which of these disclosure designs attracted the most attention when the disclosures were integrated into a media context. These studies led to the selection of the final advertising disclosure: a black rectangular graphic with the word Reclame! (i.e., Dutch for "Advertising!") in yellow letters. Finally, a two-by-two, between-subjects experimental study (disclosure design: existing versus child-inspired advertising disclosure; advertising format: brand placement versus online banner advertising) with 157 children (ages 10 and 11 years old) was performed to test the effectiveness of the child-inspired disclosure by comparing it with existing ones. This study not only showed that children recognized, understood, and liked the child-inspired disclosure better than the existing ones, but they were also better able to recognize advertising after exposure to this child-inspired advertising disclosure

Learning design – making practice explicit

New technologies have immense potential for learning, but the sheer variety possible also creates challenges for learners in terms of navigating through an increasingly complex digital landscape and for teachers in terms of how to design and support learning interventions. How can learners and teachers make informed decisions about what technologies to use in the design and support of learning activities? This presentation will consider this question and present a new methodology for design – 'learning design', which aims to shift the creation and support of learning from what has traditionally been an implicit, belief-based practice to one that is explicit and design based. Learning design research at the Open University, UK has included the development of a set of conceptual design views, a tool for visualising designs (CompendiumLD) and a social networking site, for sharing and discussing learning and teaching ideas and designs (Cloudworks). An overview of this work will be provided, along with a discussion of the perceived benefits of this new approach to educational design

Applying the interaction equivalency theorem to online courses in a large organization

Finding effective ways of designing online courses is a priority for corporate organizations. The interaction equivalency theorem states that meaningful learning can be achieved as long as courses are designed with at least a high level of one of three types of interactions (learner-content, learner-teacher or learner-learner). This study aimed to establish whether the interaction equivalency theorem applies to online learning in the corporate sector. The research was conducted in a large Mexican commercial organization, and involved 147 learners (sales supervisors), 30 teachers (sales managers and directors) and 3 academic assistants (course designers, or Education support staff). Three courses of an existing Leadership Program (Situational Leadership, Empowering Beliefs and Effective Performance) were redesigned and developed to test three course designs, each emphasizing a different type of interaction (learner-content, learner-teacher or learner-learner). Data were collected through surveys (for diagnostic and evaluation purposes) and exams. All courses yielded high levels of effectiveness, in terms of satisfaction, learning, perceived readiness for knowledge transfer and return on expectations. This suggests that the interaction equivalency theorem not only applies in a business setting but might also include other indicators of course effectiveness, such as satisfaction, learning transfer and return on expectations. Further research is needed to explore the possible expansion of the theorem

A comparison of methods for treatment selection in seamless phase II/III clinical trials incorporating information on short-term endpoints

In an adaptive seamless phase II/III clinical trial interim analysis data are used for treatment selection, enabling resources to be focussed on comparison of more effective treatment(s) with a control. In this paper we compare two methods recently proposed to enable use of short-term endpoint data for decision-making at the interim analysis. The comparison focusses on the power and the probability of correctly identifying the most promising treatment. We show that the choice of method depends on how well short-term data predict the best treatment, which may be measured by the correlation between treatment effects on short-term and long-term endpoints

Linking students' timing of engagement to learning design and academic performance

In recent years, the connection between Learning Design (LD) and Learning Analytics (LA) has been emphasized by many scholars as it could enhance our interpretation of LA findings and translate them to meaningful interventions. Together with numerous conceptual studies, a gradual accumulation of empirical evidence has indicated a strong connection between how instructors design for learning and student behaviour. Nonetheless, students' timing of engagement and its relation to LD and academic performance have received limited attention. Therefore, this study investigates to what extent students' timing of engagement aligned with instructor learning design, and how engagement varied across different levels of performance. The analysis was conducted over 28 weeks using trace data, on 387 students, and replicated over two semesters in 2015 and 2016. Our findings revealed a mismatch between how instructors designed for learning and how students studied in reality. In most weeks, students spent less time studying the assigned materials on the VLE compared to the number of hours recommended by instructors. The timing of engagement also varied, from in advance to catching up patterns. High-performing students spent more time studying in advance, while low-performing students spent a higher proportion of their time on catching-up activities. This study reinforced the importance of pedagogical context to transform analytics into actionable insights