Use of automated coding methods to assess motivational behaviour in education

Teachers’ motivational behaviour is related to important student outcomes. Assessing teachers’ motivational behaviour has been helpful to improve teaching quality and enhance student outcomes. However, researchers in educational psychology have relied on self-report or observer ratings. These methods face limitations on accurately and reliably assessing teachers’ motivational behaviour; thus restricting the pace and scale of conducting research. One potential method to overcome these restrictions is automated coding methods. These methods are capable of analysing behaviour at a large scale with less time and at low costs. In this thesis, I conducted three studies to examine the applications of an automated coding method to assess teacher motivational behaviours. First, I systematically reviewed the applications of automated coding methods used to analyse helping professionals’ interpersonal interactions using their verbal behaviour. The findings showed that automated coding methods were used in psychotherapy to predict the codes of a well-developed behavioural coding measure, in medical settings to predict conversation patterns or topics, and in education to predict simple concepts, such as the number of open/closed questions or class activity type (e.g., group work or teacher lecturing). In certain circumstances, these models achieved near human level performance. However, few studies adhered to best-practice machine learning guidelines. Second, I developed a dictionary of teachers’ motivational phrases and used it to automatically assess teachers’ motivating and de-motivating behaviours. Results showed that the dictionary ratings of teacher need support achieved a strong correlation with observer ratings of need support (rfull dictionary = .73). Third, I developed a classification of teachers’ motivational behaviour that would enable more advanced automated coding of teacher behaviours at each utterance level. In this study, I created a classification that includes 57 teacher motivating and de-motivating behaviours that are consistent with self-determination theory. Automatically assessing teachers’ motivational behaviour with automatic coding methods can provide accurate, fast pace, and large scale analysis of teacher motivational behaviour. This could allow for immediate feedback and also development of theoretical frameworks. The findings in this thesis can lead to the improvement of student motivation and other consequent student outcomes

Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues †

Research on the analysis of counselling conversations through natural language processing methods has seen remarkable growth in recent years. However, the potential of this field is still greatly limited by the lack of access to publicly available therapy dialogues, especially those with expert annotations, but it has been alleviated thanks to the recent release of AnnoMI, the first publicly and freely available conversation dataset of 133 faithfully transcribed and expert-annotated demonstrations of high- and low-quality motivational interviewing (MI)-an effective therapy strategy that evokes client motivation for positive change. In this work, we introduce new expert-annotated utterance attributes to AnnoMI and describe the entire data collection process in more detail, including dialogue source selection, transcription, annotation, and post-processing. Based on the expert annotations on key MI aspects, we carry out thorough analyses of AnnoMI with respect to counselling-related properties on the utterance, conversation, and corpus levels. Furthermore, we introduce utterance-level prediction tasks with potential real-world impacts and build baseline models. Finally, we examine the performance of the models on dialogues of different topics and probe the generalisability of the models to unseen topics