Use of automated coding methods to assess motivational behaviour in education

Teachers’ motivational behaviour is related to important student outcomes. Assessing teachers’ motivational behaviour has been helpful to improve teaching quality and enhance student outcomes. However, researchers in educational psychology have relied on self-report or observer ratings. These methods face limitations on accurately and reliably assessing teachers’ motivational behaviour; thus restricting the pace and scale of conducting research. One potential method to overcome these restrictions is automated coding methods. These methods are capable of analysing behaviour at a large scale with less time and at low costs. In this thesis, I conducted three studies to examine the applications of an automated coding method to assess teacher motivational behaviours. First, I systematically reviewed the applications of automated coding methods used to analyse helping professionals’ interpersonal interactions using their verbal behaviour. The findings showed that automated coding methods were used in psychotherapy to predict the codes of a well-developed behavioural coding measure, in medical settings to predict conversation patterns or topics, and in education to predict simple concepts, such as the number of open/closed questions or class activity type (e.g., group work or teacher lecturing). In certain circumstances, these models achieved near human level performance. However, few studies adhered to best-practice machine learning guidelines. Second, I developed a dictionary of teachers’ motivational phrases and used it to automatically assess teachers’ motivating and de-motivating behaviours. Results showed that the dictionary ratings of teacher need support achieved a strong correlation with observer ratings of need support (rfull dictionary = .73). Third, I developed a classification of teachers’ motivational behaviour that would enable more advanced automated coding of teacher behaviours at each utterance level. In this study, I created a classification that includes 57 teacher motivating and de-motivating behaviours that are consistent with self-determination theory. Automatically assessing teachers’ motivational behaviour with automatic coding methods can provide accurate, fast pace, and large scale analysis of teacher motivational behaviour. This could allow for immediate feedback and also development of theoretical frameworks. The findings in this thesis can lead to the improvement of student motivation and other consequent student outcomes

Knowledge Augmentation in Language Models to Overcome Domain Adaptation and Scarce Data Challenges in Clinical Domain

The co-existence of two scenarios, “the massive amount of unstructured text data that humanity produces” and “the scarcity of sufficient training data to train language models,” in the healthcare domain have multifold increased the need for intelligent tools and techniques to process, interpret and extract different types of knowledge from the data. My research goal in this thesis is to develop intelligent methods and models to automatically better interpret human language and sentiments, particularly its structure and semantics, to solve multiple higher-level Natural Language Processing (NLP) downstream tasks and beyond. This thesis is spread over six chapters and is divided into two parts based on the contributions. The first part is centered on best practices for modeling data and injecting domain knowledge to enrich data semantics applied to tackle several classification tasks in the healthcare domain and beyond. The contribution is to reduce the training time, improve the performance of classification models, and use world knowledge as a source of domain knowledge when working with limited/small training data. The second part introduces the one of its kind high-quality dataset of Motivational Interviewing (MI), AnnoMI, followed by the experimental benchmarking analysis for AnnoMI. The contribution accounts to provide a publicly accessible dataset of Motivational Interviewing and methods to overcome data scarcity challenges in complex domains (such as mental health). The overall organization of the thesis is as follows: \\ The first chapter provides a high-level introduction to the tools and techniques applied in the scope of the thesis. The second chapter presents optimal methods for (i) feature selection, (ii) eliminating irrelevant and superfluous attributes from the dataset, (iii) data preprocessing, and (iv) advanced data representation methods (word embedding and bag-of-words) to model data. The third chapter introduces the Language Model (LM), K-LM, a combination of Generative Pretrained Transformer (GPT)-2 and Bidirectional Encoder Representations from Transformers (BERT) that uses knowledge graphs to inject domain knowledge for domain adaptation tasks. The end goal of this chapter is to reduce the training time and improve the performance of classification models when working with limited/small training data. The fourth chapter introduces the high-quality dataset of expert-annotated MI (AnnoMI), comprised of 133 therapy session transcriptions distributed over 44 topics (including smoking cessation, anxiety management, weight loss, etc.), and provides an in-depth analysis of the dataset. \\ The fifth chapter presents the experimental analysis with AnnoMI, which includes (i) augmentation techniques to generate data and (ii) fairness and bias assessments of the employed Classical Machine Learning (CML) and Deep Learning (DL) approach to develop reliable classification models. Finally, the sixth chapter provides the conclusion and outcomes of all the work presented in this thesis. The scientific contributions of this thesis include the solution to overcome the challenges of scarce training data in complex domains and domain adaptation in LMs. The practical contributions of the thesis are data resources and the language model for a range of quantitative and qualitative NLP applications. Keywords: Natural Language Processing, Domain Adaptation, Motivational Interviewing, AI Fairness and Bias, Data Augmentation, GPT, BERT, Healthcare

The application of sentiment analysis to a psychotherapy session : an exploratory study using four general-purpose lexicons

Dissertação de Mestrado apresentada no ISPA – Instituto Universitário para obtenção de grau de Mestre na especialidade de Psicologia Clínica.In this study we explore the application of sentiment analysis to a complete and in-person psychotherapy session. Sentiment analysis is a text mining technique that allows for the analysis, interpretation, and visualization of textual data. We investigate how we can apply a lexicon-based approach to analyze clinical session data, using four general-purpose lexicons available within an open-source statistical programming language environment, R. We conducted our study by comparing the performance of four general-purpose lexicons to the performance of n = 52 human raters, using inter-rater reliability (IRR) and intraclass correlation (ICC) measurements. Our findings suggest there is low to moderate agreement between human ratings and lexicon generated ratings, depending on the lexicon used. There are some benefits in applying a lexicon-based sentiment analysis approach to psychotherapy session data, namely the way it efficiently processes and analyses data and allows for novel visualizations of psychotherapy data. We recommend further investigation into the application of sentiment analysis as a technique, focusing on the performance of specific-purpose lexicons. We also recommend further research into comparing the performance of lexicon-based approaches to text classification approaches to the analysis of psychotherapy data

Natural Language Processing for Motivational Interviewing Counselling: Addressing Challenges in Resources, Benchmarking and Evaluation

Motivational interviewing (MI) is a counselling style often used in healthcare to improve patient health and quality of life by promoting positive behaviour changes. Natural language processing (NLP) has been explored for supporting MI use cases of insights/feedback generation and therapist training, such as automatically assigning behaviour labels to therapist/client utterances and generating possible therapist responses. Despite the progress of NLP for MI applications, significant challenges remain. The most prominent one is the lack of publicly available and annotated MI dialogue corpora due to privacy constraints. Consequently, there is also a lack of common benchmarks and poor reproducibility across studies. Furthermore, human evaluation for therapist response generation is expensive and difficult to scale due to its dependence on MI experts as evaluators. In this thesis, we address these challenges in 4 directions: low-resource NLP modelling, MI dialogue dataset creation, benchmark development for real-world applicable tasks, and laypeople-experts human evaluation study. First, we explore zero-shot binary empathy assessment at the utterance level. We experiment with a supervised approach that trains on heuristically constructed empathy vs. non-empathy contrast in non-therapy dialogues. While this approach has better performance than other models without empathy-aware training, it is still suboptimal and therefore highlights the need for a well-annotated MI dataset. Next, we create AnnoMI, the first publicly available dataset of expert-annotated MI dialogues. It contains MI conversations that demonstrate both high- and low-quality counselling, with extensive annotations by domain experts covering key MI attributes. We also conduct comprehensive analyses of the dataset. Then, we investigate two AnnoMI-based real-world applicable tasks: predicting current-turn therapist/client behaviour given the utterance, and forecasting next-turn therapist behaviour given the dialogue history. We find that language models (LMs) perform well on predicting therapist behaviours with good generalisability to new dialogue topics. However, LMs have suboptimal forecasting performance, which reflects therapists' flexibility where multiple optimal next-turn actions are possible. Lastly, we ask both laypeople and experts to evaluate the generation of a crucial type of therapist responses -- reflection -- on a key quality aspect: coherence and context-consistency. We find that laypeople are a viable alternative to experts, as laypeople show good agreement with each other and correlation with experts. We also find that a large LM generates mostly coherent and consistent reflections. Overall, the work of this thesis broadens access to NLP for MI significantly as well as presents a wide range of findings on related natural language understanding/generation tasks with a real-world focus. Thus, our contributions lay the groundwork for the broader NLP community to be more engaged in research for MI, which will ultimately improve the quality of life for recipients of MI counselling

‘In my own comfort zone’: Client experiences of relational aspects of Skype therapy for alcohol problems

Background: therapists and counsellors are increasingly using Skype or equivalent video-based applications to offer treatment in place of face-to-face delivery of talking therapies. In the alcohol treatment sector, this offers a range of potential benefits including value for money, accessibility, reduced stigma and increased privacy for service users. However, the impact on the therapeutic relationship remains unclear and under-researched, particularly from a service user or client perspective.Aims: to explore how alcohol treatment clients make sense of the relational aspects of their Skype therapy, and to examine how Skype might disrupt existing ideas around the therapeutic relationship between client and therapist.Methodology: qualitative design using thematic analysis, with 15 participant interviews, conducted via Skype and telephone, with male and female adult service users from a single treatment provider. All participants undertook a minimum of four Skype treatment sessions, and had completed their therapy in the six months prior to interview.Findings: the themes that were identified highlighted the significance of the participant’s own home as the site of therapy, with emphasis on the comfort of the home, and the presence of family members and pets in the therapeutic environment. Participants stressed the importance of viewing the face of the therapist, the establishment of a therapeutic bond, and specific therapist qualities that were viewed positively. Participants also reflected on their relationship with alcohol, issues around denial and avoidance, and their decision to undertake therapy for alcohol problems ‘remotely’ via Skype.Discussion: There are nuanced and potentially unforeseen consequences of undertaking therapy for alcohol problems via Skype, relating to the significance of the therapy environment and relationship between client and therapist. It is important for treatment providers and individual practitioners to be aware of these issues in light of rapid and radical changes in the way that therapy is increasingly delivered via new video-based technologies

Measuring client modes of engagement in humanistic experiential psychotherapy

The role of clients' emotional engagement has progressively played a central role in psychotherapy. This project inserts itself in this debate by seeking to validate the Client Modes of Engagement (CME) theoretical model (Elliott 2006; 2013a). While Elliott's CME framework-a process-diagnostic map based on clients' experiential content-was grounded on decades of research and clinical practice, it had yet to be made amenable to empirical investigation.;This project responds to this absence by offering the Client Modes of Engagement Observational Coding System (CME-OCS) and the Client Modes of Engagement Questionnaire (CMEQ-R2). These instruments measure the construct from both the perspective of external observers (CME-OCS) and therapists (CMEQ-R2).;This dissertation explores the application and validation process for both the CME-OCS and the CMEQ-R2. The results confirmed that the CME-OCS is a reliable coding system for identifying CMEs during EFT psychotherapy. Additionally, the findings suggest that there are interactions between CMEs, phases of therapy, and outcome groups. Moreover, I established that there are differences in the ways outcome groups' transition between CMEs at particular stages of therapy.;I applied both classical psychometric properties methods and Rasch modelling with the purpose of examining the CMEQ-R2's psychometrics, refining the instrument, and later applying it in a process outcome study. The results suggest that levels of CME early in therapy and changes in levels of CME over therapy-as measured by the CMEQ-R2-are significantly associated with client pre-post therapeutic improvement.;I also found firm ground for arguing that therapists can distinguish between levels of CMEs and that their perspective can be systematically analysed. Together, both instruments pose important implications for research and clinical practice. Overall, this study validates the contention that researchers and therapists should be particularly attentive to clients' manner of engagement and focus of attention on specific levels of their emotion scheme.The role of clients' emotional engagement has progressively played a central role in psychotherapy. This project inserts itself in this debate by seeking to validate the Client Modes of Engagement (CME) theoretical model (Elliott 2006; 2013a). While Elliott's CME framework-a process-diagnostic map based on clients' experiential content-was grounded on decades of research and clinical practice, it had yet to be made amenable to empirical investigation.;This project responds to this absence by offering the Client Modes of Engagement Observational Coding System (CME-OCS) and the Client Modes of Engagement Questionnaire (CMEQ-R2). These instruments measure the construct from both the perspective of external observers (CME-OCS) and therapists (CMEQ-R2).;This dissertation explores the application and validation process for both the CME-OCS and the CMEQ-R2. The results confirmed that the CME-OCS is a reliable coding system for identifying CMEs during EFT psychotherapy. Additionally, the findings suggest that there are interactions between CMEs, phases of therapy, and outcome groups. Moreover, I established that there are differences in the ways outcome groups' transition between CMEs at particular stages of therapy.;I applied both classical psychometric properties methods and Rasch modelling with the purpose of examining the CMEQ-R2's psychometrics, refining the instrument, and later applying it in a process outcome study. The results suggest that levels of CME early in therapy and changes in levels of CME over therapy-as measured by the CMEQ-R2-are significantly associated with client pre-post therapeutic improvement.;I also found firm ground for arguing that therapists can distinguish between levels of CMEs and that their perspective can be systematically analysed. Together, both instruments pose important implications for research and clinical practice. Overall, this study validates the contention that researchers and therapists should be particularly attentive to clients' manner of engagement and focus of attention on specific levels of their emotion scheme