Research Methods for Education With Technology: Four Concerns, Examples, and Recommendations

The success of education with technology research is in part because the field draws upon theories and methods from multiple disciplines. However, drawing upon multiple disciplines has drawbacks because sometimes the methodological expertise of each discipline is not applied when researchers conduct studies outside of their research training. The focus here is on research using methods drawn largely from psychology, for example, evaluating the impact of different systems on how students perform. The methodological concerns discussed are: low power; not using multilevel modeling; dichotomization; and inaccurate reporting of the numeric statistics. Examples are drawn from a recent set of proceedings. Recommendations, which are applicable throughout the social sciences, are made for each of these

Can a teachable agent influence how students respond to competition in an educational game?

Learning in educational games is often associated with some form of competition. We investigated how students responded to winning or losing in an educational math game, with respect to playing with or without a Teachable Agent (TA). Students could choose between game modes in which the TA took a more passive or active role, or let the TA play a game entirely on its own. Based on the data logs from 3983 games played by 163 students (age 10–11), we analyzed data on students’ persistence, challenge-seeking and performance during gameplay. Results indicated that students showed greater persistence when playing together with the TA, by more often repeating a lost game with the TA, than a lost game after playing alone. Students’ challenge-seeking, by increasing the difficulty level, was greater following a win than following a loss, especially after the TA won on its own. Students’ gameplay performance was unaffected by their TA winning or losing but was, unexpectedly, slightly worse following a win by the student alone. We conclude that engaging a TA can make students respond more productively to both winning and losing, depending on the particular role the TA takes in the game. These results may inform more specific hypotheses as to the differential effects of competing and collaborating in novel, AI-supported social constellations, such as with TAs, on students’ motivation and ego-involvement in educational games