Beyond A/B Testing: Sequential Randomization for Developing Interventions in Scaled Digital Learning Environments

Randomized experiments ensure robust causal inference that are critical to effective learning analytics research and practice. However, traditional randomized experiments, like A/B tests, are limiting in large scale digital learning environments. While traditional experiments can accurately compare two treatment options, they are less able to inform how to adapt interventions to continually meet learners' diverse needs. In this work, we introduce a trial design for developing adaptive interventions in scaled digital learning environments -- the sequential randomized trial (SRT). With the goal of improving learner experience and developing interventions that benefit all learners at all times, SRTs inform how to sequence, time, and personalize interventions. In this paper, we provide an overview of SRTs, and we illustrate the advantages they hold compared to traditional experiments. We describe a novel SRT run in a large scale data science MOOC. The trial results contextualize how learner engagement can be addressed through inclusive culturally targeted reminder emails. We also provide practical advice for researchers who aim to run their own SRTs to develop adaptive interventions in scaled digital learning environments

Opportunities for Adaptive Experiments to Enable Continuous Improvement that Trades-off Instructor and Researcher Incentives

Randomized experimental comparisons of alternative pedagogical strategies could provide useful empirical evidence in instructors' decision-making. However, traditional experiments do not have a clear and simple pathway to using data rapidly to try to increase the chances that students in an experiment get the best conditions. Drawing inspiration from the use of machine learning and experimentation in product development at leading technology companies, we explore how adaptive experimentation might help in continuous course improvement. In adaptive experiments, as different arms/conditions are deployed to students, data is analyzed and used to change the experience for future students. This can be done using machine learning algorithms to identify which actions are more promising for improving student experience or outcomes. This algorithm can then dynamically deploy the most effective conditions to future students, resulting in better support for students' needs. We illustrate the approach with a case study providing a side-by-side comparison of traditional and adaptive experimentation of self-explanation prompts in online homework problems in a CS1 course. This provides a first step in exploring the future of how this methodology can be useful in bridging research and practice in doing continuous improvement

Improving Undergraduate Statistics Education: Educational Lessons From Two Pedagogical Experiments

The ultimate goal of statistics education is to create a statistically literate society in which people can appropriately use statistical thinking. Although the need to improve the teaching of introductory statistics courses is not a new one, with increased demand on these courses, there has been constant effort to seek out better ways of teaching these courses. The University of Kentucky (UK) began a reform of its general education program in November 2005. Thinking and reasoning are the central themes of this well-designed general education curriculum. The main goal of this dissertation is to fill in some gaps in the research literature on the teaching and learning of statistics. This dissertation includes two independent studies (experiments). The first study will examine the instructional effects of physical versus virtual manipulatives (see definitions later) on learning outcomes in introductory statistics, whereas the second study will investigate the impact of different styles in teaching statistics (inverted classroom versus traditional classroom) on learning outcomes in introductory statistics. In general, this dissertation strives to join many other reform efforts to explore instructional ways that engage students in reasoning and thinking statistically. To combat the abstract nature of probability and statistics, the use of manipulatives may represent one of the most effective strategies in the statistics classroom. There are fundamental reasons to inherently value the inverted classroom’s emphasis on activity-based learning and increased responsibility of the students to become active participants in their own learning. The results of the first study revealed that there were no significant differences between the business as usual group who received traditional concrete manipulatives and the experimental group who received online virtual manipulatives. There -were no statistically significant interaction effects between types of manipulatives and high school ACT mathematics scores, informing the literature that ability levels neither intensify nor weaken the effects of types of manipulatives. The results of the study did not show a significant difference in GPA one year later between the experimental group and the business as usual group. The results of the second study revealed that there were some significant differences between the business as usual group who received traditional lecture type classroom and the experimental group who received inverted. We compared all seven outcomes for the two groups: projects average, tests average, classwork, midterm attendance average, class final attendance average, midterm grade and class final grade. Students in the traditional classroom did better than students in the inverted classroom in projects average, classwork, midterm attendance average, midterm grade and class final grade. We used tree different blocks with student background variables as predictors. The first one, individual student background, is explained by age, gender and ethnicity. High school background variables is explained by high school GPA and ACT mathematics scores. The third one, university program background, is explained by university cumulative GPA and student major. After controlling for student background variables, students in the traditional classroom did better than students in the inverted classroom in projects average, overall classwork and midterm grade. The model when controlling for student high school background variables showed that students in the traditional classroom did better than students in the inverted classroom in projects average, overall classwork and midterm grade. Finally, after controlling for student university background variables, students in the traditional classroom performed similarly to students in the inverted classroom in projects average, test average, overall classwork, midterm attendance average, class final attendance, midterm grade and class final grade. When controlling for all (i.e., student background variables, student high school background variables, and university program variables), students in the traditional classroom did better than students in the inverted classroom in midterm grade only. The results of the study may not be generalized to the population of all undergraduate students. It also gives no indication of how the results would generalize to other content domains. Further studies may explore along these lines of inquiry regarding the effects of virtual manipulatives in comparison with concrete manipulatives and the effects of the traditional classroom in comparison with inverted. Further studies may seek some longer period of using and comparing the two teaching methods

A Virtual Community of Practice to Introduce Evidence-based Pedagogy in Chemical, Materials, and Biological Engineering Courses

This paper describes a model for a virtual community of practice (VCP) to support faculty efforts to adopt research-based instructional strategies in Chemical, Materials and Biological Engineering courses. The VCP was built on published recommendations for successful faculty development programs. The VCP program began with a 10 week virtual training period for five pairs of VCP leaders, during which they acquired the skills and knowledge needed to lead the faculty VCP. The faculty VCPs focused on one of five technical disciplines and were led by a pair of leaders having expertise in a specific technical focus area as well as in engineering pedagogy. Workshops were held using Internet conferencing software: the first 8 weekly workshops provided training in research-based pedagogy, and the second 8 biweekly workshops supported faculty efforts to implement chosen strategies in their courses. The participants were full-time faculty members with a range of teaching experience and pedagogical expertise, ranging from novice to expert. Improvement was measured via pre/post survey in the areas of familiarity and use of research-based pedagogy, as well as in perceived student motivation. The second part of the paper focuses on the translation of faculty participant experiences from the VCP into the classroom as they implemented a variety of instructional methods in their courses. We describe their approaches and preliminary results using different instructional methods such as flipping the classroom, using game-based pedagogy, promoting positive interdependence in cooperative-learning teams, peer instruction, small group discussion, Process Oriented Guided Inquiry Learning (POGIL), and using Bloom’s Taxonomy to structure a course

Critical Components of Formative Assessment in Process-Oriented Guided Inquiry Learning for Online Labs

In the traditional lab setting, it is reasonably straightforward to monitor student learning and provide ongoing feedback. Such formative assessments can help students identify their strengths and weaknesses, and assist faculty to recognize where students are struggling and address problems immediately. But in an online virtual lab setting, formative assessment has challenges that go beyond space-time synchrony of online classroom. As we see increased enrollment in online courses, learning science needs to address the problem of formative assessment in online laboratory sessions. We developed a student team learning monitor (STLM module) in an electronic health record system to measure student engagement and actualize the social constructivist approach of Process Oriented Guided Inquiry Learning (POGIL). Using iterative Plan-Do-Study-Act cycles in two undergraduate courses over a period of two years, we identified critical components that are required for online implementation of POGIL. We reviewed published research on POGIL classroom implementations for the last ten years and identified some common elements that affect learning gains. We present the critical components that are necessary for implementing POGIL in online lab settings, and refer to this as Cyber POGIL. Incorporating these critical components are required to determine when, how and the circumstances under which Cyber POGIL may be successfully implemented. We recommend that more online tools be developed for POGIL classrooms, which evolve from just providing synchronous communication to improved task monitoring and assistive feedback

Enhanced online course design and its effect on the perceived level of community of inquiry

Academic institutions increasingly have adopted the online platform due to its low delivery cost and ease of scalability to large numbers of students. The pressure to increase enrollment numbers without enhancements to online course design have created the problem of lower retention and completion rates which can effect institutional funding. The purpose of this study was to explore the effect of enhancements to course design as well as class size on the level of perceived Community of Inquiry (CoI) experienced by college students. The primary research question was; what effect does enhanced online course design have on the perceived level of CoI among college students? A secondary research question was; what effect does class size have on the perceived level of CoI among online college students? The theoretical framework that informed this study was Community of Inquiry developed by Garrison (2000). This study employed a quasi-experimental research design since subjects were already enrolled in course sections. Cluster random sampling method was employed to select both the non-enhanced and enhanced class sections. The researcher surveyed subjects using a 34 question 5-scale summated CoI instrument including teacher, social, and cognitive presence. The population from which the sample was derived consisted of undergraduate college students over the age of 18 years old of any gender enrolled in at least one completely online 16-week class at the OSU-OKC campus. The researcher employed One-Way MANOVA and Pearson r correlation inferential statistical analysis to test all research hypotheses. The findings indicate that there is no evidence of significant effect between enhanced course design and the perceived level of CoI among college students. The findings regarding correlation of CoI scores and class size however showed there was a strong negative correlation between teaching presence, a moderate positive correlation between social presence, and a weak negative correlation between cognitive presence and class size. Other factors such as teacher training, facilitation by the instructor, student readiness, and the type of course taught and its effect to perceived levels of CoI might be considered for future research

Defining Active Learning: A Restricted Systematic Review

What is active learning? While active learning has been demonstrated to have positive impacts on student learning and performance, defining the concept has been elusive. Previous research examining active learning definitions in STEM fields found that the vast majority of published articles did not define active learning, and those that did defined active learning as interacting, engaging, or not lecturing. The current research extends this STEM-focused work by examining both social science and STEM science publications. A restricted systematic review of literature was conducted using the SCOPUS database, resulting in 547 relevant articles focused on active learning from 2017 to 2022. An examination of the articles indicated that 71% of the reviewed articles did not define active learning and that the instructional strategies most often cited as fostering active learning emphasized social interactive learning strategies (e.g., small groups, team-based learning, discussion, and cooperative learning), as well as critical thinking strategies (e.g., problem-based learning, case-based learning, and inquiry-based learning). In addition, an in-depth qualitative analysis of the 161 definitions provided within the articles yielded three main emergent themes: (a) active learning is defined as grounded in student-centered constructivist theory, (b) active learning is defined as promoting higher-order thinking and deep learning, and (c) active learning is defined as an instructional strategy involving activity, participation, and engagement. Given these main findings, a representative definition was created: Active learning is a student-centered approach to the construction of knowledge focused on activities and strategies that foster higher-order thinking. Click here to read the corresponding ISSOTL blog post

A Meta-Analysis of Teacher and Student-Centered Practices and Processes in Undergraduate Science Education

This meta-analysis investigates the effects of four instructional dimensions rated on a scale from more Teacher-centered (T-C) to more Student-centered (S-C) plus several coded moderator variables on the achievement of undergraduate students in science education courses. More student-centered conditions served as the ‘treatment’ while more teacher-centered conditions were considered the ‘control.’ Hedges’ g, operationalized as the adjusted standardized differences between treatment and control means, served as the outcome measure. The weighted average difference between groups was g̅ = 0.34, k = 140 (random effects analysis), indicating an overall difference in favor of student-centered instruction. Out of four rated dimensions (Pacing, Teacher’s Role, Flexibility, and Adaptation) only Flexibility was significant in metaregression as a negative predictor of effect size. Two demographic variables (i.e., class size & subject matter), and one instructional moderator variables (i.e., technology use) were also significant when added to Flexibility, producing a model that accounted for 36% of total variation in effect size

ClassNet: a potential computer-mediated communications learning tool in preservice teacher education?

This study investigates computer-mediated communications (CMC) in preservice teacher education. Three papers respectively examine current CMC uses and outcomes, describe a new CMC tool which may enhance future use, and present an evaluation of that tool with preservice teachers;The first paper reviews the utilization of CMC in the preservice curriculum and then analyzes occurring problems and assesses resulting needs. The problems are divided into two broad categories: Those that may dissipate with improvement of existing CMC technology and those that may persist even with improved technology. A study of these problems reveals that learning from CMC integration must be more closely investigated and that additional CMC tools must be created to support new and potentially valuable learning experiences. One such tool, ClassNet, is described, and its supported learning opportunities are illustrated;The second paper explains in detail the nature of ClassNet. This paper begins by discussing the emerging problem of virtual classroom management and introduces ClassNet as a plausible solution to that problem. Then, a description of ClassNet\u27s use is given from both an instructor\u27s and a student\u27s perspective and assignment concepts are discussed. In closing, the paper provides concrete examples of ClassNet\u27s use in various subject domains and summarizes ClassNet\u27s features and future prospects;Finally, the third paper presents a qualitative evaluation of a ClassNet feature used with preservice teachers. This feature allows teachers to analyze and guide student interactions (protocols) with Java simulations. Subsequently, its value was explored in a virtual partnership of preservice teachers and distant eighth grade students joined by an online mathematics graphing simulation. Primary themes showed that preservice teachers valued the experience, constructed knowledge of student thinking, and experienced instruction central to mathematics reform. Overall, the feature is considered feasible to use in situations where a few preservice teachers are closely monitored by supervising teachers, but its utilization in a large preservice classroom is questioned. Future research includes a need to investigate different participants, different settings, and different simulations. Additionally, the development and evaluation of a synchronous component is needed, and the design of additional protocol exploration activities is advocated