Challenges of using observational data to determine the importance of example usage

Educational interventions are often evaluated with randomized control trials, which can be very expensive to conduct. One of the promises of “Big Data” in education is to use non-experimental data to discover insights. We focus on studying the impact of example usage in a Java programming tutoring system using observational data. For this, we compare different formulations of a recently proposed generalized Knowledge Tracing framework called FAST. We discover that different formulations can have the same predictive performance; yet their coefficients may have opposite signs, which may lead researchers to contradictory conclusions. We discuss implications of using fully data-driven approaches to study non-experimental data