TOBB-ETU at CLEF 2019: Prioritizing claims based on check-worthiness

20th Working Notes of CLEF Conference and Labs of the Evaluation Forum, CLEF ( 2019: Lugano; Switzerland)In recent years, we witnessed an incredible amount of misinformation spread over the Internet. However, it is extremely time consuming to analyze the veracity of every claim made on the Internet. Thus, we urgently need automated systems that can prioritize claims based on their check-worthiness, helping fact-checkers to focus on important claims. In this paper, we present our hybrid approach which combines rule-based and supervised methods for CLEF-2019 Check That! Lab's Check-Worthiness task. Our primary model ranked 9th based on MAP, and 6th based on R-P, P@5, and P@20 metrics in the official evaluation of primary submissions. © 2019 CEUR-WS. All rights reserved

LLVMs4Protest: Harnessing the Power of Large Language and Vision Models for Deciphering Protests in the News

Large language and vision models have transformed how social movements scholars identify protest and extract key protest attributes from multi-modal data such as texts, images, and videos. This article documents how we fine-tuned two large pretrained transformer models, including longformer and swin-transformer v2, to infer potential protests in news articles using textual and imagery data. First, the longformer model was fine-tuned using the Dynamic of Collective Action (DoCA) Corpus. We matched the New York Times articles with the DoCA database to obtain a training dataset for downstream tasks. Second, the swin-transformer v2 models was trained on UCLA-protest imagery data. UCLA-protest project contains labeled imagery data with information such as protest, violence, and sign. Both fine-tuned models will be available via \url{https://github.com/Joshzyj/llvms4protest}. We release this short technical report for social movement scholars who are interested in using LLVMs to infer protests in textual and imagery data

Early Detection of Depression and Eating Disorders in Spanish: UNSL at MentalRiskES 2023

MentalRiskES is a novel challenge that proposes to solve problems related to early risk detection for the Spanish language. The objective is to detect, as soon as possible, Telegram users who show signs of mental disorders considering different tasks. Task 1 involved the users' detection of eating disorders, Task 2 focused on depression detection, and Task 3 aimed at detecting an unknown disorder. These tasks were divided into subtasks, each one defining a resolution approach. Our research group participated in subtask A for Tasks 1 and 2: a binary classification problem that evaluated whether the users were positive or negative. To solve these tasks, we proposed models based on Transformers followed by a decision policy according to criteria defined by an early detection framework. One of the models presented an extended vocabulary with important words for each task to be solved. In addition, we applied a decision policy based on the history of predictions that the model performs during user evaluation. For Tasks 1 and 2, we obtained the second-best performance according to rankings based on classification and latency, demonstrating the effectiveness and consistency of our approaches for solving early detection problems in the Spanish language.Comment: In Iberian Languages Evaluation Forum (IberLEF 2023), Ja\'en, Spai

Overview of ImageCLEFcoral 2019 task

Understanding the composition of species in ecosystems on a large scale is key to developing effective solutions for marine conservation, hence there is a need to classify imagery automatically and rapidly. In 2019, ImageCLEF proposed for the first time the ImageCLEFcoral task. The task requires participants to automatically annotate and localize benthic substrate (such as hard coral, soft coral, algae and sponge) in a collection of images originating from a growing, large-scale dataset from coral reefs around the world as part of monitoring programmes. In its first edition, five groups participated submitting 20 runs using a variety of machine learning and deep learning approaches. Best runs achieved 0.24 in the annotation and localisation subtask and 0.04 on the pixel-wise parsing subtask in terms of MAP 0.5 IoU scores which measures the Mean Average Precision (MAP) when using the performance measure of Intersection over Union (IoU) bigger to 0.5 of the ground truth

Making sense of nonsense : Integrated gradient-based input reduction to improve recall for check-worthy claim detection

Analysing long text documents of political discourse to identify check-worthy claims (claim detection) is known to be an important task in automated fact-checking systems, as it saves the precious time of fact-checkers, allowing for more fact-checks. However, existing methods use black-box deep neural NLP models to detect check-worthy claims, which limits the understanding of the model and the mistakes they make. The aim of this study is therefore to leverage an explainable neural NLP method to improve the claim detection task. Specifically, we exploit well known integrated gradient-based input reduction on textCNN and BiLSTM to create two different reduced claim data sets from ClaimBuster. We observe that a higher recall in check-worthy claim detection is achieved on the data reduced by BiLSTM compared to the models trained on claims. This is an important remark since the cost of overlooking check-worthy claims is high in claim detection for fact-checking. This is also the case when a pre-trained BERT sequence classification model is fine-tuned on the reduced data set. We argue that removing superfluous tokens using explainable NLP could unlock the true potential of neural language models for claim detection, even though the reduced claims might make no sense to humans. Our findings provide insights on task formulation, design of annotation schema and data set preparation for check-worthy claim detection.publishedVersio

Strategies to Harness the Transformers' Potential: UNSL at eRisk 2023

The CLEF eRisk Laboratory explores solutions to different tasks related to risk detection on the Internet. In the 2023 edition, Task 1 consisted of searching for symptoms of depression, the objective of which was to extract user writings according to their relevance to the BDI Questionnaire symptoms. Task 2 was related to the problem of early detection of pathological gambling risks, where the participants had to detect users at risk as quickly as possible. Finally, Task 3 consisted of estimating the severity levels of signs of eating disorders. Our research group participated in the first two tasks, proposing solutions based on Transformers. For Task 1, we applied different approaches that can be interesting in information retrieval tasks. Two proposals were based on the similarity of contextualized embedding vectors, and the other one was based on prompting, an attractive current technique of machine learning. For Task 2, we proposed three fine-tuned models followed by decision policy according to criteria defined by an early detection framework. One model presented extended vocabulary with important words to the addressed domain. In the last task, we obtained good performances considering the decision-based metrics, ranking-based metrics, and runtime. In this work, we explore different ways to deploy the predictive potential of Transformers in eRisk tasks.Comment: In Conference and Labs of the Evaluation Forum (CLEF 2023), Thessaloniki, Greec

ChatGPT in the Classroom:A Preliminary Exploration on the Feasibility of Adapting ChatGPT to Support Children's Information Discovery

The influence of ChatGPT and similar models on education is being increasingly discussed. With the current level of enthusiasm among users, ChatGPT is envisioned as having great potential. As generative models are unpredictable in terms of producing biased, harmful, and unsafe content, we argue that they should be comprehensively tested for more vulnerable groups, such as children, to understand what role they can play and what training and supervision are necessary. Here, we present the results of a preliminary exploration aiming to understand whether ChatGPT can adapt to support children in completing information discovery tasks in the education context. We analyze ChatGPT responses to search prompts related to the 4th grade classroom curriculum using a variety of lenses (e.g., readability and language) to identify open challenges and limitations that must be addressed by interdisciplinary communities. Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Web Information System

Detección anticipada de riesgos en la web

Este artículo describe, brevemente, las tareas de investigación que nuestro grupo está llevando a cabo en el área de Detección Anticipada de Riesgos (DAR) en la Web. Esta línea de investigación comenzó en el año 2017 con la participación de nuestro grupo en la tarea eRisk 2017: Pilot Task on Early Detection of Depression donde se obtuvo el mejor desempeño (de acuerdo a la medida ERDE50) sobre un total de 30 contribuciones de 8 instituciones diferentes de Francia, Alemania, USA, México, Argentina, Canadá y Rusia. A patir de ese evento, se continuó participando en forma ininterrumpida en este evento en otras tareas de DAR vinculadas a depresión, anorexia, y auto-lesiones con distintos enfoques surgidos de los trabajos de postgrado de 5 tesistas de Maestría y Doctorado. En todas las participaciones del grupo, se han presentado propuestas que consituyen en la actualidad el estado del arte del área con más de 12 publicaciones en el tema.Red de Universidades con Carreras en Informátic

An Asynchronous Scheme for the Distributed Evaluation of Interactive Multimedia Retrieval

Evaluation campaigns for interactive multimedia retrieval, such as the Video Browser Shodown (VBS) or the Lifelog Search Challenge (LSC), so far imposed constraints on both simultaneity and locality of all participants, requiring them to solve the same tasks in the same place, at the same time and under the same conditions. These constraints are in contrast to other evaluation campaigns that do not focus on interactivity, where participants can process the tasks in any place at any time. The recent travel restrictions necessitated the relaxation of the locality constraint of interactive campaigns, enabling participants to take place from an arbitrary location. Born out of necessity, this relaxation turned out to be a boon since it greatly simplified the evaluation process and enabled organisation of ad-hoc evaluations outside of the large campaigns. However, it also introduced an additional complication in cases where participants were spread over several time zones. In this paper, we introduce an evaluation scheme for interactive retrieval evaluation that relaxes both the simultaneity and locality constraints, enabling participation from any place at any time within a predefined time frame. This scheme, as implemented in the Distributed Retrieval Evaluation Server (DRES), enables novel ways of conducting interactive retrieval evaluation and bridged the gap between interactive campaigns and non-interactive ones