Multiple-Choice Question Answering Models for Automatic Depression Severity Estimation

Presented at the 4th XoveTIC Conference, A Coruña, Spain, 7–8 October 2021.[Abstract] Depression is one of the most prevalent mental health diseases. Although there are effective treatments, the main problem relies on providing early and effective risk detection. Medical experts use self-reporting questionnaires to elaborate their diagnosis, but these questionnaires have some limitations. Social stigmas and the lack of awareness often negatively affect the success of these self-report questionnaires. This article aims to describe techniques to automatically estimate the depression severity from users on social media. We explored the use of pre-trained language models over the subject’s writings. We addressed the task “Measuring the Severity of the Signs of Depression” of eRisk 2020, an initiative in the CLEF Conference. In this task, participants have to fill the Beck Depression Questionnaire (BDI-II). Our proposal explores the application of pre-trained Multiple-Choice Question Answering (MCQA) models to predict user’s answers to the BDI-II questionnaire using their posts on social media. These MCQA models are built over the BERT (Bidirectional Encoder Representations from Transformers) architecture. Our results showed that multiple-choice question answering models could be a suitable alternative for estimating the depression degree, even when small amounts of training data are available (20 users).This work was supported by projects RTI2018-093336-B-C22 (MCIU & ERDF), GPC ED431B 2019/03 (Xunta de Galicia & ERDF) and CITIC, which is financial supported by Consellería de Educación, Universidade e Formación Profesional of the Xunta de Galicia through the ERDF (80%) and Secretaría Xeral de Universidades (20%), (Ref ED431G 2019/01).Xunta de Galicia; ED431B 2019/03Xunta de Galicia; ED431G 2019/0

Extração de informação de saúde através das redes sociais

Social media has been proven to be an excellent resource for connecting people and creating a parallel community. Turning it into a suitable source for extracting real world events information and information about its users as well. All of this information can be carefully re-arranged for social monitoring purposes and for the good of its community. For extracting health evidence in the social media, we started by analyzing and identifying postpartum depression in social media posts. We participated in an online challenge, eRisk 2020, continuing the previous participation of BioInfo@UAVR, predicting self-harm users based on their publications on Reddit. We built an algorithm based on methods of Natural Language Processing capable of pre-processing text data and vectorizing it. We make use of linguistic features based on the frequency of specific sets of words, and other models widely used that represent whole documents with vectors, such as Tf-Idf and Doc2Vec. The vectors and the correspondent label are then passed to a Machine Learning classifier in order to train it. Based on the patterns it found, the model predicts a classification for unlabeled users. We use multiple classifiers, to find the one that behaves the best with the data. With the goal of getting the most out of the model, an optimization step is performed in which we remove stop words and set the text vectorization algorithms and classifier to be ran in parallel. An analysis of the feature importance is integrated and a validation step is performed. The results are discussed and presented in various plots, and include a comparison between different tuning strategies and the relation between the parameters and the score. We conclude that the choice of parameters is essential for achieving a better score and for finding them, there are other strategies more efficient then the widely used Grid Search. Finally, we compare several approaches for building an incremental classification based on the post timeline of the users. And conclude that it is possible to have a chronological perception of certain traits of Reddit users, specifically evaluating the risk of self-harm with a F1 Score of 0.73.As redes sociais são um excelente recurso para conectar pessoas, criando assim uma comunidade paralela em que fluem informações acerca de eventos globais bem como sobre os seus utilizadores. Toda esta informação pode ser trabalhada com o intuito de monitorizar o bem estar da sua comunidade. De forma a encontrar evidência médica nas redes sociais, começámos por analisar e identificar posts de mães em risco de depressão pós-parto no Reddit. Participámos num concurso online, eRisk 2020, com o intuito de continuar a participação da equipa BioInfo@ UAVR, em que prevemos utilizadores que estão em risco de se automutilarem através da análise das suas publicações no Reddit. Construímos um algoritmo com base em métodos de Processamento de Linguagem Natural capaz de pré-processar os dados de texto e vectorizá-los. Fazendo uso de características linguísticas baseadas na frequência de conjuntos de palavras, e outros modelos usados globalmente, capazes de representar documentos com vetores, como o Tf-Idf e o Doc2Vec. Os vetores e a sua respetiva classificação são depois disponibilizados a algoritmos de Aprendizagem Automática, para serem treinados e encontrar padrões entre eles. Utilizamos vários classificadores, de forma a encontrar o que se comporta melhor com os dados. Com base nos padrões que encontrou, os classificadores prevêm a classificação de utilizadores ainda por avaliar. De forma a tirar o máximo proveito do algoritmo, é desempenhada uma otimização em que as stop words são removidas e paralelizamos os algoritmos de vectorização de texto e o classificador. Incorporamos uma análise da importância dos atributos do modelo e a otimização dos híper parâmetros de forma a obter um resultado melhor. Os resultados são discutidos e apresentados em múltiplos plots, e incluem a comparação entre diferentes estratégias de optimização e observamos a relação entre os parâmetros e a sua performance. Concluimos que a escolha dos parâmetros é essencial para conseguir melhores resultados e que para os encontrar, existem estratégias mais eficientes que o habitual Grid Search, como o Random Search e a Bayesian Optimization. Comparamos também várias abordagens para formar uma classificação incremental que tem em conta a cronologia dos posts. Concluimos que é possível ter uma perceção cronológica de traços dos utilizadores do Reddit, nomeadamente avaliar o risco de automutilação, com um F1 Score de 0,73.Mestrado em Engenharia de Computadores e Telemátic

CeDRI at eRisk 2021: a naive approach to early detection of psychological disorders in social media

This paper describes the participation of the CeDRI team in eRisk 2021 tasks, particularly, the Task 1: Early Detection of Signs of Pathological Gambling and Task 2: Early Detection of Signs of Self-Harm. The main difference between these two is that the first is a “test only” challenge, where no training data is supplied. The second task has labeled data available, which can be used for training. Both tasks were addressed using the same algorithms, using a custom training set for Task 1 and the provided data in the second. The algorithms were TfIdf vectorizer with a Logistic Regression layer, Word2Vec vectorizer with LSTM and Word2Vec vectorizer with CNN. All vectorizers and Neural Networks were trained solely with the training data. As expected, the algorithms did not state-of-the-art, but the experience allowed to reflect in several aspects related to the importance of proper dataset preparation and processing. © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).info:eu-repo/semantics/publishedVersio

Semantic Similarity Models for Depression Severity Estimation

Depressive disorders constitute a severe public health issue worldwide. However, public health systems have limited capacity for case detection and diagnosis. In this regard, the widespread use of social media has opened up a way to access public information on a large scale. Computational methods can serve as support tools for rapid screening by exploiting this user-generated social media content. This paper presents an efficient semantic pipeline to study depression severity in individuals based on their social media writings. We select test user sentences for producing semantic rankings over an index of representative training sentences corresponding to depressive symptoms and severity levels. Then, we use the sentences from those results as evidence for predicting users' symptom severity. For that, we explore different aggregation methods to answer one of four Beck Depression Inventory (BDI) options per symptom. We evaluate our methods on two Reddit-based benchmarks, achieving 30\% improvement over state of the art in terms of measuring depression severity.Comment: Accepted at the EMNLP 2023 conferenc

Improving the Generalizability of Depression Detection by Leveraging Clinical Questionnaires

Automated methods have been widely used to identify and analyze mental health conditions (e.g., depression) from various sources of information, including social media. Yet, deployment of such models in real-world healthcare applications faces challenges including poor out-of-domain generalization and lack of trust in black box models. In this work, we propose approaches for depression detection that are constrained to different degrees by the presence of symptoms described in PHQ9, a questionnaire used by clinicians in the depression screening process. In dataset-transfer experiments on three social media datasets, we find that grounding the model in PHQ9's symptoms substantially improves its ability to generalize to out-of-distribution data compared to a standard BERT-based approach. Furthermore, this approach can still perform competitively on in-domain data. These results and our qualitative analyses suggest that grounding model predictions in clinically-relevant symptoms can improve generalizability while producing a model that is easier to inspect

Improving the Generalizability of Depression Detection by Leveraging Clinical Questionnaires

Automated methods have been widely used to identify and analyze mental healthconditions (e.g., depression) from various sources of information, includingsocial media. Yet, deployment of such models in real-world healthcareapplications faces challenges including poor out-of-domain generalization andlack of trust in black box models. In this work, we propose approaches fordepression detection that are constrained to different degrees by the presenceof symptoms described in PHQ9, a questionnaire used by clinicians in thedepression screening process. In dataset-transfer experiments on three socialmedia datasets, we find that grounding the model in PHQ9's symptomssubstantially improves its ability to generalize to out-of-distribution datacompared to a standard BERT-based approach. Furthermore, this approach canstill perform competitively on in-domain data. These results and ourqualitative analyses suggest that grounding model predictions inclinically-relevant symptoms can improve generalizability while producing amodel that is easier to inspect.<br

BDI-Sen: a sentence dataset for clinical symptoms of depression

[Abstract] People tend to consider social platforms as convenient media for expressing their concerns and emotional struggles. With their widespread use, researchers could access and analyze user-generated content related to mental states. Computational models that exploit that data show promising results in detecting at-risk users based on engineered features or deep learning models. However, recent works revealed that these approaches have a limited capacity for generalization and interpretation when considering clinical settings. Grounding the models' decisions on clinical and recognized symptoms can help to overcome these limitations. In this paper, we introduce BDI-Sen, a symptom-annotated sentence dataset for depressive disorder. BDI-Sen covers all the symptoms present in the Beck Depression Inventory-II (BDI-II), a reliable questionnaire used for detecting and measuring depression. The annotations in the collection reflect whether a statement about the specific symptom is informative (i.e., exposes traces about the individual's state regarding that symptom). We thoroughly analyze this resource and explore linguistic style, emotional attribution, and other psycholinguistic markers. Additionally, we conduct a series of experiments investigating the utility of BDI-Sen for various tasks, including the detection and severity classification of symptoms. We also examine their generalization when considering symptoms from other mental diseases. BDI-Sen may aid the development of future models that consider trustworthy and valuable depression markers

Strategies to Harness the Transformers' Potential: UNSL at eRisk 2023

The CLEF eRisk Laboratory explores solutions to different tasks related to risk detection on the Internet. In the 2023 edition, Task 1 consisted of searching for symptoms of depression, the objective of which was to extract user writings according to their relevance to the BDI Questionnaire symptoms. Task 2 was related to the problem of early detection of pathological gambling risks, where the participants had to detect users at risk as quickly as possible. Finally, Task 3 consisted of estimating the severity levels of signs of eating disorders. Our research group participated in the first two tasks, proposing solutions based on Transformers. For Task 1, we applied different approaches that can be interesting in information retrieval tasks. Two proposals were based on the similarity of contextualized embedding vectors, and the other one was based on prompting, an attractive current technique of machine learning. For Task 2, we proposed three fine-tuned models followed by decision policy according to criteria defined by an early detection framework. One model presented extended vocabulary with important words to the addressed domain. In the last task, we obtained good performances considering the decision-based metrics, ranking-based metrics, and runtime. In this work, we explore different ways to deploy the predictive potential of Transformers in eRisk tasks.Comment: In Conference and Labs of the Evaluation Forum (CLEF 2023), Thessaloniki, Greec