LEAN-LIFE: A Label-Efficient Annotation Framework Towards Learning from Explanation

Successfully training a deep neural network demands a huge corpus of labeled data. However, each label only provides limited information to learn from and collecting the requisite number of labels involves massive human effort. In this work, we introduce LEAN-LIFE, a web-based, Label-Efficient AnnotatioN framework for sequence labeling and classification tasks, with an easy-to-use UI that not only allows an annotator to provide the needed labels for a task, but also enables LearnIng From Explanations for each labeling decision. Such explanations enable us to generate useful additional labeled data from unlabeled instances, bolstering the pool of available training data. On three popular NLP tasks (named entity recognition, relation extraction, sentiment analysis), we find that using this enhanced supervision allows our models to surpass competitive baseline F1 scores by more than 5-10 percentage points, while using 2X times fewer labeled instances. Our framework is the first to utilize this enhanced supervision technique and does so for three important tasks -- thus providing improved annotation recommendations to users and an ability to build datasets of (data, label, explanation) triples instead of the regular (data, label) pair.Comment: Accepted to the ACL 2020 (demo). The first two authors contributed equally. Project page: http://inklab.usc.edu/leanlife

Social media mental health analysis framework through applied computational approaches

Studies have shown that mental illness burdens not only public health and productivity but also established market economies throughout the world. However, mental disorders are difficult to diagnose and monitor through traditional methods, which heavily rely on interviews, questionnaires and surveys, resulting in high under-diagnosis and under-treatment rates. The increasing use of online social media, such as Facebook and Twitter, is now a common part of people’s everyday life. The continuous and real-time user-generated content often reflects feelings, opinions, social status and behaviours of individuals, creating an unprecedented wealth of person-specific information. With advances in data science, social media has already been increasingly employed in population health monitoring and more recently mental health applications to understand mental disorders as well as to develop online screening and intervention tools. However, existing research efforts are still in their infancy, primarily aimed at highlighting the potential of employing social media in mental health research. The majority of work is developed on ad hoc datasets and lacks a systematic research pipeline. [Continues.]</div

Predicting the Need for Urgent Instructor Intervention in MOOC Environments

In recent years, massive open online courses (MOOCs) have become universal knowledge resources and arguably one of the most exciting innovations in e-learning environments. MOOC platforms comprise numerous courses covering a wide range of subjects and domains. Thousands of learners around the world enrol on these online platforms to satisfy their learning needs (mostly) free of charge. However, the retention rates of MOOC courses (i.e., those who successfully complete a course of study) are low (around 10% on average); dropout rates tend to be very high (around 90%). The principal channel via which MOOC learners can communicate their difficulties with the learning content and ask for assistance from instructors is by posting in a dedicated MOOC forum. Importantly, in the case of learners who are suffering from burnout or stress, some of these posts require urgent intervention. Given the above, urgent instructor intervention regarding learner requests for assistance via posts made on MOOC forums has become an important topic for research among researchers. Timely intervention by MOOC instructors may mitigate dropout issues and make the difference between a learner dropping out or staying on a course. However, due to the typically extremely high learner-to-instructor ratio in MOOCs and the often-huge numbers of posts on forums, while truly urgent posts are rare, managing them can be very challenging –– if not sometimes impossible. Instructors can find it challenging to monitor all existing posts and identify which posts require immediate intervention to help learners, encourage retention, and reduce the current high dropout rates. The main objective of this research project, therefore, was thus to mine and analyse learners’ MOOC posts as a fundamental step towards understanding their need for instructor intervention. To achieve this, the researcher proposed and built comprehensive classification models to predict the need for instructor intervention. The ultimate goal is to help instructors by guiding them to posts, topics, and learners that require immediate interventions. Given the above research aim the researcher conducted different experiments to fill the gap in literature based on different platform datasets (the FutureLearn platform and the Stanford MOOCPosts dataset) in terms of the former, three MOOC corpora were prepared: two of them gold-standard MOOC corpora to identify urgent posts, annotated by selected experts in the field; the third is a corpus detailing learner dropout. Based in these datasets, different architectures and classification models based on traditional machine learning, and deep learning approaches were proposed. In this thesis, the task of determining the need for instructor intervention was tackled from three perspectives: (i) identifying relevant posts, (ii) identifying relevant topics, and (iii) identifying relevant learners. Posts written by learners were classified into two categories: (i) (urgent) intervention and (ii) (non-urgent) intervention. Also, learners were classified into: (i) requiring instructor intervention (at risk of dropout) and (ii) no need for instructor intervention (completer). In identifying posts, two experiments were used to contribute to this field. The first is a novel classifier based on a deep learning model that integrates novel MOOC post dimensions such as numerical data in addition to textual data; this represents a novel contribution to the literature as all available models at the time of writing were based on text-only. The results demonstrate that the combined, multidimensional features model proposed in this project is more effective than the text-only model. The second contribution relates to creating various simple and hybrid deep learning models by applying plug & play techniques with different types of inputs (word-based or word-character-based) and different ways of representing target input words as vector representations of a particular word. According to the experimental findings, employing Bidirectional Encoder Representations from Transformers (BERT) for word embedding rather than word2vec as the former is more effective at the intervention task than the latter across all models. Interestingly, adding word-character inputs with BERT does not improve performance as it does for word2vec. Additionally, on the task of identifying topics, this is the first time in the literature that specific language terms to identify the need for urgent intervention in MOOCs were obtained. This was achieved by analysing learner MOOC posts using latent Dirichlet allocation (LDA) and offers a visualisation tool for instructors or learners that may assist them and improve instructor intervention. In addition, this thesis contributes to the literature by creating mechanisms for identifying MOOC learners who may need instructor intervention in a new context, i.e., by using their historical online forum posts as a multi-input approach for other deep learning architectures and Transformer models. The findings demonstrate that using the Transformer model is more effective at identifying MOOC learners who require instructor intervention. Next, the thesis sought to expand its methodology to identify posts that relate to learner behaviour, which is also a novel contribution, by proposing a novel priority model to identify the urgency of intervention building based on learner histories. This model can classify learners into three groups: low risk, mid risk, and high risk. The results show that the completion rates of high-risk learners are very low, which confirms the importance of this model. Next, as MOOC data in terms of urgent posts tend to be highly unbalanced, the thesis contributes by examining various data balancing methods to spot situations in which MOOC posts urgently require instructor assistance. This included developing learner and instructor models to assist instructors to respond to urgent MOOCs posts. The results show that models with undersampling can predict the most urgent cases; 3x augmentation + undersampling usually attains the best performance. Finally, for the first time, this thesis contributes to the literature by applying text classification explainability (eXplainable Artificial Intelligence (XAI)) to an instructor intervention model, demonstrating how using a reliable predictor in combination with XAI and colour-coded visualisation could be utilised to assist instructors in deciding when posts require urgent intervention, as well as supporting annotators to create high-quality, gold-standard datasets to determine posts cases where urgent intervention is required

On the Promotion of the Social Web Intelligence

Given the ever-growing information generated through various online social outlets, analytical research on social media has intensified in the past few years from all walks of life. In particular, works on social Web intelligence foster and benefit from the wisdom of the crowds and attempt to derive actionable information from such data. In the form of collective intelligence, crowds gather together and contribute to solving problems that may be difficult or impossible to solve by individuals and single computers. In addition, the consumer insight revealed from social footprints can be leveraged to build powerful business intelligence tools, enabling efficient and effective decision-making processes. This dissertation is broadly concerned with the intelligence that can emerge from the social Web platforms. In particular, the two phenomena of social privacy and online persuasion are identified as the two pillars of the social Web intelligence, studying which is essential in the promotion and advancement of both collective and business intelligence. The first part of the dissertation is focused on the phenomenon of social privacy. This work is mainly motivated by the privacy dichotomy problem. Users often face difficulties specifying privacy policies that are consistent with their actual privacy concerns and attitudes. As such, before making use of social data, it is imperative to employ multiple safeguards beyond the current privacy settings of users. As a possible solution, we utilize user social footprints to detect their privacy preferences automatically. An unsupervised collaborative filtering approach is proposed to characterize the attributes of publicly available accounts that are intended to be private. Unlike the majority of earlier studies, a variety of social data types is taken into account, including the social context, the published content, as well as the profile attributes of users. Our approach can provide support in making an informed decision whether to exploit one\u27s publicly available data to draw intelligence. With the aim of gaining insight into the strategies behind online persuasion, the second part of the dissertation studies written comments in online deliberations. Specifically, we explore different dimensions of the language, the temporal aspects of the communication, as well as the attributes of the participating users to understand what makes people change their beliefs. In addition, we investigate the factors that are perceived to be the reasons behind persuasion by the users. We link our findings to traditional persuasion research, hoping to uncover when and how they apply to online persuasion. A set of rhetorical relations is known to be of importance in persuasive discourse. We further study the automatic identification and disambiguation of such rhetorical relations, aiming to take a step closer towards automatic analysis of online persuasion. Finally, a small proof of concept tool is presented, showing the value of our persuasion and rhetoric studies

Argumentation dialogues in web-based GDSS: an approach using machine learning techniques

Tese de doutoramento em InformaticsA tomada de decisão está presente no dia a dia de qualquer pessoa, mesmo que muitas vezes ela não tenha consciência disso. As decisões podem estar relacionadas com problemas quotidianos, ou podem estar relacionadas com questões mais complexas, como é o caso das questões organizacionais. Normalmente, no contexto organizacional, as decisões são tomadas em grupo. Os Sistemas de Apoio à Decisão em Grupo têm sido estudados ao longo das últimas décadas com o objetivo de melhorar o apoio prestado aos decisores nas mais diversas situações e/ou problemas a resolver. Existem duas abordagens principais à implementação de Sistemas de Apoio à Decisão em Grupo: a abordagem clássica, baseada na agregação matemática das preferências dos diferentes elementos do grupo e as abordagens baseadas na negociação automática (e.g. Teoria dos Jogos, Argumentação, entre outras). Os atuais Sistemas de Apoio à Decisão em Grupo baseados em argumentação podem gerar uma enorme quantidade de dados. O objetivo deste trabalho de investigação é estudar e desenvolver modelos utilizando técnicas de aprendizagem automática para extrair conhecimento dos diálogos argumentativos realizados pelos decisores, mais concretamente, pretende-se criar modelos para analisar, classificar e processar esses dados, potencializando a geração de novo conhecimento que será utilizado tanto por agentes inteligentes, como por decisiores reais. Promovendo desta forma a obtenção de consenso entre os membros do grupo. Com base no estudo da literatura e nos desafios em aberto neste domínio, formulou-se a seguinte hipótese de investigação - É possível usar técnicas de aprendizagem automática para apoiar diálogos argumentativos em Sistemas de Apoio à Decisão em Grupo baseados na web. No âmbito dos trabalhos desenvolvidos, foram aplicados algoritmos de classificação supervisionados a um conjunto de dados contendo argumentos extraídos de debates online, criando um classificador de frases argumentativas que pode classificar automaticamente (A favor/Contra) frases argumentativas trocadas no contexto da tomada de decisão. Foi desenvolvido um modelo de clustering dinâmico para organizar as conversas com base nos argumentos utilizados. Além disso, foi proposto um Sistema de Apoio à Decisão em Grupo baseado na web que possibilita apoiar grupos de decisores independentemente de sua localização geográfica. O sistema permite a criação de problemas multicritério e a configuração das preferências, intenções e interesses de cada decisor. Este sistema de apoio à decisão baseado na web inclui os dashboards de relatórios inteligentes que são gerados através dos resultados dos trabalhos alcançados pelos modelos anteriores já referidos. A concretização de cada um dos objetivos permitiu validar as questões de investigação identificadas e assim responder positivamente à hipótese definida.Decision-making is present in anyone’s daily life, even if they are often unaware of it. Decisions can be related to everyday problems, or they can be related to more complex issues, such as organizational issues. Normally, in the organizational context, decisions are made in groups. Group Decision Support Systems have been studied over the past decades with the aim of improving the support provided to decision-makers in the most diverse situations and/or problems to be solved. There are two main approaches to implementing Group Decision Support Systems: the classical approach, based on the mathematical aggregation of the preferences of the different elements of the group, and the approaches based on automatic negotiation (e.g. Game Theory, Argumentation, among others). Current argumentation-based Group Decision Support Systems can generate an enormous amount of data. The objective of this research work is to study and develop models using automatic learning techniques to extract knowledge from argumentative dialogues carried out by decision-makers, more specifically, it is intended to create models to analyze, classify and process these data, enhancing the generation of new knowledge that will be used both by intelligent agents and by real decision-makers. Promoting in this way the achievement of consensus among the members of the group. Based on the literature study and the open challenges in this domain, the following research hypothesis was formulated - It is possible to use machine learning techniques to support argumentative dialogues in web-based Group Decision Support Systems. As part of the work developed, supervised classification algorithms were applied to a data set containing arguments extracted from online debates, creating an argumentative sentence classifier that can automatically classify (For/Against) argumentative sentences exchanged in the context of decision-making. A dynamic clustering model was developed to organize conversations based on the arguments used. In addition, a web-based Group Decision Support System was proposed that makes it possible to support groups of decision-makers regardless of their geographic location. The system allows the creation of multicriteria problems and the configuration of preferences, intentions, and interests of each decision-maker. This web-based decision support system includes dashboards of intelligent reports that are generated through the results of the work achieved by the previous models already mentioned. The achievement of each objective allowed validation of the identified research questions and thus responded positively to the defined hypothesis.I also thank to Fundação para a Ciência e a Tecnologia, for the Ph.D. grant funding with the reference: SFRH/BD/137150/2018

Functional Text Dimensions for the annotation of web corpora

This paper presents an approach to classifying large web corpora into genres by means of Functional Text Dimensions (FTDs). This offers a topological approach to text typology in which the texts are described in terms of their similarity to prototype genres. The suggested set of categories is designed to be applicable to any text on the web and to be reliable in annotation practice. Interannotator agreement results show that the suggested categories produce Krippendorff's α at above 0.76. In addition to the functional space of eighteen dimensions, similarity between annotated documents can be described visually within a space of reduced dimensions obtained through t-distributed Statistical Neighbour Embedding. Reliably annotated texts also provide the basis for automatic genre classification, which can be done in each FTD, as well as as within the space of reduced dimensions. An example comparing texts from the Brown Corpus, the BNC and ukWac, a large web corpus, is provided

NLP-Based Techniques for Cyber Threat Intelligence

In the digital era, threat actors employ sophisticated techniques for which, often, digital traces in the form of textual data are available. Cyber Threat Intelligence~(CTI) is related to all the solutions inherent to data collection, processing, and analysis useful to understand a threat actor's targets and attack behavior. Currently, CTI is assuming an always more crucial role in identifying and mitigating threats and enabling proactive defense strategies. In this context, NLP, an artificial intelligence branch, has emerged as a powerful tool for enhancing threat intelligence capabilities. This survey paper provides a comprehensive overview of NLP-based techniques applied in the context of threat intelligence. It begins by describing the foundational definitions and principles of CTI as a major tool for safeguarding digital assets. It then undertakes a thorough examination of NLP-based techniques for CTI data crawling from Web sources, CTI data analysis, Relation Extraction from cybersecurity data, CTI sharing and collaboration, and security threats of CTI. Finally, the challenges and limitations of NLP in threat intelligence are exhaustively examined, including data quality issues and ethical considerations. This survey draws a complete framework and serves as a valuable resource for security professionals and researchers seeking to understand the state-of-the-art NLP-based threat intelligence techniques and their potential impact on cybersecurity

Enhancing Hate Speech Detection in Sinhala Language on Social Media using Machine Learning

To counter the harmful dissemination of hate speech on social media, especially abusive outbursts of racism and sexism, automatic and accurate detection is crucial. However, a significant challenge lies in the vast sparsity of available data, hindering accurate classification. This study presents a novel approach to Sinhala hate speech detection on social platforms by coupling a global feature selection process with traditional machine learning, the research scrutinizes hate speech intricacies. A class-based variable feature selection process evaluates significance via global and local scores, identifying optimal values for prevalent classifiers. Utilizing class-based and corpus-based evaluations, we pinpoint optimal feature values for classifiers like SVM, MNB, and RF. Our results reveal notable enhancements in performance, specifically the F1-Score, underscoring how feature selection and parameter tuning work in tandem to boost model efficacy. Furthermore, the study explores nuanced variations in classifier performance across training and testing datasets, emphasizing the importance of model generalization