Embedding Experiments: Staking Causal Inference in Authentic Educational Contexts

To identify the ways teachers and educational systems can improve learning, researchers need to make causal inferences. Analyses of existing datasets play an important role in detecting causal patterns, but conducting experiments also plays an indispensable role in this research. In this article, we advocate for experiments to be embedded in real educational contexts, allowing researchers to test whether interventions such as a learning activity, new technology, or advising strategy elicit reliable improvements in authentic student behaviours and educational outcomes. Embedded experiments, wherein theoretically relevant variables are systematically manipulated in real learning contexts, carry strong benefits for making causal inferences, particularly when allied with the data-rich resources of contemporary e-learning environments. Toward this goal, we offer a field guide to embedded experimentation, reviewing experimental design choices, addressing ethical concerns, discussing the importance of involving teachers, and reviewing how interventions can be deployed in a variety of contexts, at a range of scales. Causal inference is a critical component of a field that aims to improve student learning; including experimentation alongside analyses of existing data in learning analytics is the most compelling way to test causal claims

Self-regulated studying behavior, and the social norms that influence it

Teachers commonly use injunctive norms when telling students what they should be doing. But researchers find that sometimes descriptive norms, information about what others are actually doing, are more powerful influencers of behavior. In the present work, we examine which norm is more effective at increasing self‐regulated studying and performance in an online college course across two semesters. To do this, we randomly assigned 751 undergraduate Introductory Psychology students to receive email messages at the start of every content unit that either contained descriptive norms, injunctive norms, information about the course, or a no message control. We found that injunctive norms increased study behaviors aimed at fulfilling course requirements (completion of assigned activities), but did not improve learning outcomes. Descriptive norms increased behaviors aimed at improving knowledge (ungraded practice with activities after they were due), and improved performance. These results suggest that norms more effectively influence behavior when there is a match, or a sense of fit, between the goal of the behavior (fulfilling course requirements vs. learning) and the pull of a stated norm (social approval vs. efficacy). We discuss these implications with respect to students' motivations for self‐regulated studying behavior in contemporary learning environments, and the overall goals of education