Beyond A/B Testing: Sequential Randomization for Developing Interventions in Scaled Digital Learning Environments

Randomized experiments ensure robust causal inference that are critical to effective learning analytics research and practice. However, traditional randomized experiments, like A/B tests, are limiting in large scale digital learning environments. While traditional experiments can accurately compare two treatment options, they are less able to inform how to adapt interventions to continually meet learners' diverse needs. In this work, we introduce a trial design for developing adaptive interventions in scaled digital learning environments -- the sequential randomized trial (SRT). With the goal of improving learner experience and developing interventions that benefit all learners at all times, SRTs inform how to sequence, time, and personalize interventions. In this paper, we provide an overview of SRTs, and we illustrate the advantages they hold compared to traditional experiments. We describe a novel SRT run in a large scale data science MOOC. The trial results contextualize how learner engagement can be addressed through inclusive culturally targeted reminder emails. We also provide practical advice for researchers who aim to run their own SRTs to develop adaptive interventions in scaled digital learning environments

Personalizing the design of computer‐based instruction to enhance learning

This paper reports two studies designed to investigate the effect on learning outcomes of matching individuals’ preferred cognitive styles to computer‐based instructional (CBI) material. Study 1 considered the styles individually as Verbalizer, Imager, Wholist and Analytic. Study 2 considered the bi‐dimensional nature of cognitive styles in order to assess the full ramification of cognitive styles on learning: Analytic/Imager, Analytic/ Verbalizer, Wholist/Imager and the Wholist/Verbalizer. The mix of images and text, the nature of the text material, use of advance organizers and proximity of information to facilitate meaningful connections between various pieces of information were some of the considerations in the design of the CBI material. In a quasi‐experimental format, students’ cognitive styles were analysed by Cognitive Style Analysis (CSA) software. On the basis of the CSA result, the system defaulted students to either matched or mismatched CBI material by alternating between the two formats. The instructional material had a learning and a test phase. Learning outcome was tested on recall, labelling, explanation and problem‐solving tasks. Comparison of the matched and mismatched instruction did not indicate significant difference between the groups, but the consistently better performance by the matched group suggests potential for further investigations where the limitations cited in this paper are eliminated. The result did indicate a significant difference between the four cognitive styles with the Wholist/Verbalizer group performing better then all other cognitive styles. Analysing the difference between cognitive styles on individual test tasks indicated significant difference on recall, labelling and explanation, suggesting that certain test tasks may suit certain cognitive styles

The Serums Tool-Chain:Ensuring Security and Privacy of Medical Data in Smart Patient-Centric Healthcare Systems

Digital technology is permeating all aspects of human society and life. This leads to humans becoming highly dependent on digital devices, including upon digital: assistance, intelligence, and decisions. A major concern of this digital dependence is the lack of human oversight or intervention in many of the ways humans use this technology. This dependence and reliance on digital technology raises concerns in how humans trust such systems, and how to ensure digital technology behaves appropriately. This works considers recent developments and projects that combine digital technology and artificial intelligence with human society. The focus is on critical scenarios where failure of digital technology can lead to significant harm or even death. We explore how to build trust for users of digital technology in such scenarios and considering many different challenges for digital technology. The approaches applied and proposed here address user trust along many dimensions and aim to build collaborative and empowering use of digital technologies in critical aspects of human society

Applying machine learning to predict future adherence to physical activity programs.

BackgroundIdentifying individuals who are unlikely to adhere to a physical exercise regime has potential to improve physical activity interventions. The aim of this paper is to develop and test adherence prediction models using objectively measured physical activity data in the Mobile Phone-Based Physical Activity Education program (mPED) trial. To the best of our knowledge, this is the first to apply Machine Learning methods to predict exercise relapse using accelerometer-recorded physical activity data.MethodsWe use logistic regression and support vector machine methods to design two versions of a Discontinuation Prediction Score (DiPS), which uses objectively measured past data (e.g., steps and goal achievement) to provide a numerical quantity indicating the likelihood of exercise relapse in the upcoming week. The respective prediction accuracy of these two versions of DiPS are compared, and then numerical simulation is performed to explore the potential of using DiPS to selectively allocate financial incentives to participants to encourage them to increase physical activity.Resultswe had access to a physical activity trial data that were continuously collected every 60 sec every day for 9 months in 210 participants. By using the first 15 weeks of data as training and test on weeks 16-30, we show that both versions of DiPS have a test AUC of 0.9 with high sensitivity and specificity in predicting the probability of exercise adherence. Simulation results assuming different intervention regimes suggest the potential benefit of using DiPS as a score to allocate resources in physical activity intervention programs in reducing costs over other allocation schemes.ConclusionsDiPS is capable of making accurate and robust predictions for future weeks. The most predictive features are steps and physical activity intensity. Furthermore, the use of DiPS scores can be a promising approach to determine when or if to provide just-in-time messages and step goal adjustments to improve compliance. Further studies on the use of DiPS in the design of physical activity promotion programs are warranted.Trial registrationClinicalTrials.gov NCT01280812 Registered on January 21, 2011

Improving School Improvement

PREFACEIn opening this volume, you might be thinking:Is another book on school improvement really needed?Clearly our answer is yes. Our analyses of prevailing school improvement legislation, planning, and literature indicates fundamental deficiencies, especially with respect to enhancing equity of opportunity and closing the achievement gap.Here is what our work uniquely brings to policy and planning tables:(1) An expanded framework for school improvement – We highlight that moving from a two- to a three-component policy and practice framework is essential for closing the opportunity and achievement gaps. (That is, expanding from focusing primarily on instruction and management/government concerns by establishing a third primary component to improve how schools address barriers to learning and teaching.)(2) An emphasis on integrating a deep understanding of motivation – We underscore that concerns about engagement, management of behavior, school climate, equity of opportunity, and student outcomes require an up-to-date grasp of motivation and especially intrinsic motivation.(3) Clarification of the nature and scope of personalized teaching – We define personalization as the process of matching learner motivation and capabilities and stress that it is the learner's perception that determines whether the match is a good one.(4) A reframing of remediation and special education – We formulate these processes as personalized special assistance that is applied in and out of classrooms and practiced in a sequential and hierarchical manner.(5) A prototype for transforming student and learning supports – We provide a framework for a unified, comprehensive, and equitable system designed to address barriers to learning and teaching and re-engage disconnected students and families.(6) A reworking of the leadership structure for whole school improvement --We outline how the operational infrastructure can and must be realigned in keeping with a three component school improvement framework.(7) A systemic approach to enhancing school-community collaboration – We delineate a leadership role for schools in outreaching to communities in order to work on shared concerns through a formal collaborative operational infrastructure that enables weaving together resources to advance the work.(8) An expanded framework for school accountability – We reframe school accountability to ensure a balanced approach that accounts for a shift to a three component school improvement policy.(9) Guidance for substantive, scalable, and sustainable systemic changes –We frame mechanisms and discuss lessons learned related to facilitating fundamental systemic changes and replicating and sustaining them across a district.The frameworks and practices presented are based on our many years of work in schools and from efforts to enhance school-community collaboration. We incorporate insights from various theories and the large body of relevant research and from lessons learned and shared by many school leaders and staff who strive everyday to do their best for children.Our emphasis on new directions in no way is meant to demean current efforts. We know that the demands placed on those working in schools go well beyond what anyone should be asked to do. Given the current working conditions in many schools, our intent is to help make the hard work generate better results. To this end, we highlight new directions and systemic pathways for improving school outcomes.Some of what we propose is difficult to accomplish. Hopefully, the fact that there are schools, districts, and state agencies already trailblazing the way will engender a sense of hope and encouragement to those committed to innovation.It will be obvious that our work owes much to many. We are especially grateful to those who are pioneering major systemic changes across the country. These leaders and so many in the field have generously offered their insights and wisdom. And, of course, we are indebted to hundreds of scholars whose research and writing is a shared treasure. As always, we take this opportunity to thank Perry Nelson and the host of graduate and undergraduate students at UCLA who contribute so much to our work each day, and to the many young people and their families who continue to teach us all.Respectfully submitted for your consideration,Howard Adelman & Linda Taylo