Reflection and Learning Robustness in a Natural Language Conceptual Physics Tutoring System

This thesis investigates whether reflection after tutoring with the Itspoke qualitative physics tutoring system can improve both near and far transfer learning and retention. This question is formalized in three major hypotheses. H1: that reading a post-tutoring reflective text will improve learning compared to reading a non-reflective text. H2: that a more cohesive reflective text will produce higher learning gains for most students. And H3: that students with high domain knowledge will learn more from a less cohesive text.In addition, this thesis addresses the question of which mechanisms affect learning from a reflective text. Secondary hypotheses H4 and H5 posit that textual cohesion and student motivation, respectively, each affect learning by influencing the amount of inference performed while reading.These hypotheses were tested by asking students to read a reflective/abstractive text after tutoring with the Itspoke tutor. This text compared dialog parts in which similar physics principles had been applied to different situations. Students were randomly assigned among two experimental conditions which got ``high' or ``low' cohesion versions of this text, or a control condition which read non-reflective physics material after tutoring.The secondary hypotheses were tested using two measures of cognitive load while reading: reading speeds and a self-report measure of reading difficulty.Near and far transfer learning was measured using sets of questions that were mostly isomorphic vs. non-isomorphic the tutored problems, and retention was measured by administering both an immediate and a delayed post-test. Motivation was measured using a questionnaire.Reading a reflective text improved learning, but only for students with a middle amount of motivation, confirming H1 for that group. These students also learned more from a more cohesive reflective text, supporting H2. Cohesion also affected high and low knowledge students significantly differently, supporting H3, except that high knowledge students learned best from high, not low cohesion text.Students with higher amounts of motivation did have higher cognitive load, confirming hypothesis H5 and suggesting that they engaged the text more actively. However, secondary hypothesis H4 failed to show a role for cognitive load in explaining the learning interaction between knowledge and cohesion demonstrated in H3

Computational Argumentation Approaches to Improve Sensemaking and Evidence-based Reasoning in Online Deliberation Systems

Deliberation is the process through which communities identify potential solutions for a problem and select the solution that most effectively meets their diverse requirements through dialogic communication. Online deliberation is implemented nowadays with means of social media and online discussion platforms; however, these media present significant challenges and issues that can be traced to inadequate support for Sensemaking processes and poor endorsement of the quality characteristics of deliberation. This thesis investigates integrating computational argumentation methods in online deliberation platforms as an effective way to improve participants' perception of the quality of the deliberation process, their way of making sense of the overall process and producing healthier social dynamics. For that, two computational artefacts are proposed: (i) a Synoptical summariser of long discussions and (ii) a Scientific Argument Recommender System (SciArgRecSys). The two artefacts are designed and developed with state-of-the-art methods (with the use of Large Language Models - LLMs) and evaluated intrinsically and extrinsically when deployed in a real live platform (BCause). Through extensive evaluation, the positive effect of both artefacts is illustrated in human Sensemaking and essential quality characteristics of deliberation such as reciprocal Engagement, Mutual Understanding, and Social dynamics. In addition, it has been demonstrated that these interventions effectively reduce polarisation, the formation of sub-communities while significantly enhancing the quality of the discussion by making it more coherent and diverse

Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation

This paper surveys the current state of the art in Natural Language Generation (NLG), defined as the task of generating text or speech from non-linguistic input. A survey of NLG is timely in view of the changes that the field has undergone over the past decade or so, especially in relation to new (usually data-driven) methods, as well as new applications of NLG technology. This survey therefore aims to (a) give an up-to-date synthesis of research on the core tasks in NLG and the architectures adopted in which such tasks are organised; (b) highlight a number of relatively recent research topics that have arisen partly as a result of growing synergies between NLG and other areas of artificial intelligence; (c) draw attention to the challenges in NLG evaluation, relating them to similar challenges faced in other areas of Natural Language Processing, with an emphasis on different evaluation methods and the relationships between them.Comment: Published in Journal of AI Research (JAIR), volume 61, pp 75-170. 118 pages, 8 figures, 1 tabl

AI approaches to understand human deceptions, perceptions, and perspectives in social media

Social media platforms have created virtual space for sharing user generated information, connecting, and interacting among users. However, there are research and societal challenges: 1) The users are generating and sharing the disinformation 2) It is difficult to understand citizens\u27 perceptions or opinions expressed on wide variety of topics; and 3) There are overloaded information and echo chamber problems without overall understanding of the different perspectives taken by different people or groups. This dissertation addresses these three research challenges with advanced AI and Machine Learning approaches. To address the fake news, as deceptions on the facts, this dissertation presents Machine Learning approaches for fake news detection models, and a hybrid method for topic identification, whether they are fake or real. To understand the user\u27s perceptions or attitude toward some topics, this study analyzes the sentiments expressed in social media text. The sentiment analysis of posts can be used as an indicator to measure how topics are perceived by the users and how their perceptions as a whole can affect decision makers in government and industry, especially during the COVID-19 pandemic. It is difficult to measure the public perception of government policies issued during the pandemic. The citizen responses to the government policies are diverse, ranging from security or goodwill to confusion, fear, or anger. This dissertation provides a near real-time approach to track and monitor public reactions toward government policies by continuously collecting and analyzing Twitter posts about the COVID-19 pandemic. To address the social media\u27s overwhelming number of posts, content echo-chamber, and information isolation issue, this dissertation provides a multiple view-based summarization framework where the same contents can be summarized according to different perspectives. This framework includes components of choosing the perspectives, and advanced text summarization approaches. The proposed approaches in this dissertation are demonstrated with a prototype system to continuously collect Twitter data about COVID-19 government health policies and provide analysis of citizen concerns toward the policies, and the data is analyzed for fake news detection and for generating multiple-view summaries

Improving Neural Question Answering with Retrieval and Generation

Text-based Question Answering (QA) is a subject of interest both for its practical applications, and as a test-bed to measure the key Artificial Intelligence competencies of Natural Language Processing (NLP) and the representation and application of knowledge. QA has progressed a great deal in recent years by adopting neural networks, the construction of large training datasets, and unsupervised pretraining. Despite these successes, QA models require large amounts of hand-annotated data, struggle to apply supplied knowledge effectively, and can be computationally ex- pensive to operate. In this thesis, we employ natural language generation and information retrieval techniques in order to explore and address these three issues. We first approach the task of Reading Comprehension (RC), with the aim of lifting the requirement for in-domain hand-annotated training data. We describe a method for inducing RC capabilities without requiring hand-annotated RC instances, and demonstrate performance on par with early supervised approaches. We then explore multi-lingual RC, and develop a dataset to evaluate methods which enable training RC models in one language, and testing them in another. Second, we explore open-domain QA (ODQA), and consider how to build mod- els which best leverage the knowledge contained in a Wikipedia text corpus. We demonstrate that retrieval-augmentation greatly improves the factual predictions of large pretrained language models in unsupervised settings. We then introduce a class of retrieval-augmented generator model, and demonstrate its strength and flexibility across a range of knowledge-intensive NLP tasks, including ODQA. Lastly, we study the relationship between memorisation and generalisation in ODQA, developing a behavioural framework based on memorisation to contextualise the performance of ODQA models. Based on these insights, we introduce a class of ODQA model based on the concept of representing knowledge as question- answer pairs, and demonstrate how, by using question generation, such models can achieve high accuracy, fast inference, and well-calibrated predictions

Demonstration-based help for interactive systems

The usability of day-to-day applications is of utmost importance. The lack of usability of these is one of the causes of frustration at work, as it creates barriers to the execution of tasks. The Change of applications by third parties, to increase usability, is difficult because it requires, usually, access to source codes and an increase on its complexity. This work proposes and implements a demonstration help tool that allows the improvement of tasks completion, decreases the time spent, and reduces the cost of learning. An analysis of work on aid tools is presented identifying positive aspects and research opportunities. The help tool developed allows the creation of automation through picture-driven computing, which makes it possible to develop help mechanisms independent from application source codes. Since the tool is image oriented, and tasks can involve multiple applications, it is also possible to develop help scripts that are not restricted to just one application. User studies were done with the objectives of validating the work developed and identifying platforms and tasks with usability problems in the business world. It was concluded that the work has positive effects in the accomplishment of tasks.A usabilidade das aplicações utilizadas no dia-a-dia é de extrema importância. A falta de usabilidade destas é um dos causadores de frustração no trabalho, pois cria barreiras à execução de tarefas. A alteração das aplicações por terceiros de forma a aumentar a usabilidade é difícil pois requer, usualmente, acesso aos códigos fonte e incremento da sua complexidade. Este trabalho propõe e implementa uma ferramenta de ajuda por demonstração que visa melhorar o sucesso na realização de tarefas, reduzir o tempo despendido, e diminuir o esforço de aprendizagem. Uma análise a trabalhos sobre ferramentas de ajuda é apresentada identificando aspetos positivos e oportunidades de investigação. A ferramenta de ajuda desenvolvida permite a criação de automações através de computação por imagem, que tornam possível o desenvolvimento de mecanismos de ajuda independentes dos códigos fonte das aplicações. Sendo que a ferramenta é orientada a imagem, e que as tarefas podem envolver múltiplas aplicações, torna-se também possível o desenvolvimento de scripts de ajuda não restringidos a apenas uma aplicação. Foram realizados estudos com utilizadores com os objetivos de validar o trabalho desenvolvido e identificar plataformas e tarefas com problemas de usabilidade no meio empresarial. Deste modo, concluiu-se que o trabalho tem efeitos positivos na realização de tarefas

Natural Language Processing for Motivational Interviewing Counselling: Addressing Challenges in Resources, Benchmarking and Evaluation

Motivational interviewing (MI) is a counselling style often used in healthcare to improve patient health and quality of life by promoting positive behaviour changes. Natural language processing (NLP) has been explored for supporting MI use cases of insights/feedback generation and therapist training, such as automatically assigning behaviour labels to therapist/client utterances and generating possible therapist responses. Despite the progress of NLP for MI applications, significant challenges remain. The most prominent one is the lack of publicly available and annotated MI dialogue corpora due to privacy constraints. Consequently, there is also a lack of common benchmarks and poor reproducibility across studies. Furthermore, human evaluation for therapist response generation is expensive and difficult to scale due to its dependence on MI experts as evaluators. In this thesis, we address these challenges in 4 directions: low-resource NLP modelling, MI dialogue dataset creation, benchmark development for real-world applicable tasks, and laypeople-experts human evaluation study. First, we explore zero-shot binary empathy assessment at the utterance level. We experiment with a supervised approach that trains on heuristically constructed empathy vs. non-empathy contrast in non-therapy dialogues. While this approach has better performance than other models without empathy-aware training, it is still suboptimal and therefore highlights the need for a well-annotated MI dataset. Next, we create AnnoMI, the first publicly available dataset of expert-annotated MI dialogues. It contains MI conversations that demonstrate both high- and low-quality counselling, with extensive annotations by domain experts covering key MI attributes. We also conduct comprehensive analyses of the dataset. Then, we investigate two AnnoMI-based real-world applicable tasks: predicting current-turn therapist/client behaviour given the utterance, and forecasting next-turn therapist behaviour given the dialogue history. We find that language models (LMs) perform well on predicting therapist behaviours with good generalisability to new dialogue topics. However, LMs have suboptimal forecasting performance, which reflects therapists' flexibility where multiple optimal next-turn actions are possible. Lastly, we ask both laypeople and experts to evaluate the generation of a crucial type of therapist responses -- reflection -- on a key quality aspect: coherence and context-consistency. We find that laypeople are a viable alternative to experts, as laypeople show good agreement with each other and correlation with experts. We also find that a large LM generates mostly coherent and consistent reflections. Overall, the work of this thesis broadens access to NLP for MI significantly as well as presents a wide range of findings on related natural language understanding/generation tasks with a real-world focus. Thus, our contributions lay the groundwork for the broader NLP community to be more engaged in research for MI, which will ultimately improve the quality of life for recipients of MI counselling

Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021

The eighth edition of the Italian Conference on Computational Linguistics (CLiC-it 2021) was held at Università degli Studi di Milano-Bicocca from 26th to 28th January 2022. After the edition of 2020, which was held in fully virtual mode due to the health emergency related to Covid-19, CLiC-it 2021 represented the first moment for the Italian research community of Computational Linguistics to meet in person after more than one year of full/partial lockdown