One-Shot Labeling for Automatic Relevance Estimation

Dealing with unjudged documents ("holes") in relevance assessments is a perennial problem when evaluating search systems with offline experiments. Holes can reduce the apparent effectiveness of retrieval systems during evaluation and introduce biases in models trained with incomplete data. In this work, we explore whether large language models can help us fill such holes to improve offline evaluations. We examine an extreme, albeit common, evaluation setting wherein only a single known relevant document per query is available for evaluation. We then explore various approaches for predicting the relevance of unjudged documents with respect to a query and the known relevant document, including nearest neighbor, supervised, and prompting techniques. We find that although the predictions of these One-Shot Labelers (1SL) frequently disagree with human assessments, the labels they produce yield a far more reliable ranking of systems than the single labels do alone. Specifically, the strongest approaches can consistently reach system ranking correlations of over 0.86 with the full rankings over a variety of measures. Meanwhile, the approach substantially increases the reliability of t-tests due to filling holes in relevance assessments, giving researchers more confidence in results they find to be significant. Alongside this work, we release an easy-to-use software package to enable the use of 1SL for evaluation of other ad-hoc collections or systems.Comment: SIGIR 202

RSDD-Time: Temporal Annotation of Self-Reported Mental Health Diagnoses

Self-reported diagnosis statements have been widely employed in studying language related to mental health in social media. However, existing research has largely ignored the temporality of mental health diagnoses. In this work, we introduce RSDD-Time: a new dataset of 598 manually annotated self-reported depression diagnosis posts from Reddit that include temporal information about the diagnosis. Annotations include whether a mental health condition is present and how recently the diagnosis happened. Furthermore, we include exact temporal spans that relate to the date of diagnosis. This information is valuable for various computational methods to examine mental health through social media because one's mental health state is not static. We also test several baseline classification and extraction approaches, which suggest that extracting temporal information from self-reported diagnosis statements is challenging.Comment: 6 pages, accepted for publication at the CLPsych workshop at NAACL-HLT 201

A Neural Attention Model for Categorizing Patient Safety Events

Medical errors are leading causes of death in the US and as such, prevention of these errors is paramount to promoting health care. Patient Safety Event reports are narratives describing potential adverse events to the patients and are important in identifying and preventing medical errors. We present a neural network architecture for identifying the type of safety events which is the first step in understanding these narratives. Our proposed model is based on a soft neural attention model to improve the effectiveness of encoding long sequences. Empirical results on two large-scale real-world datasets of patient safety reports demonstrate the effectiveness of our method with significant improvements over existing methods.Comment: ECIR 201

Characterizing Question Facets for Complex Answer Retrieval

Complex answer retrieval (CAR) is the process of retrieving answers to questions that have multifaceted or nuanced answers. In this work, we present two novel approaches for CAR based on the observation that question facets can vary in utility: from structural (facets that can apply to many similar topics, such as 'History') to topical (facets that are specific to the question's topic, such as the 'Westward expansion' of the United States). We first explore a way to incorporate facet utility into ranking models during query term score combination. We then explore a general approach to reform the structure of ranking models to aid in learning of facet utility in the query-document term matching phase. When we use our techniques with a leading neural ranker on the TREC CAR dataset, our methods rank first in the 2017 TREC CAR benchmark, and yield up to 26% higher performance than the next best method.Comment: 4 pages; SIGIR 2018 Short Pape

La gestione dei rifiuti fra strumenti di "command and control" e strumenti economici e finanziari

Nel lavoro di tesi ho analizzato i rimedi giuridici applicati, o potenzialmente applicabili, per garantire una corretta gestione dei rifiuti e, allo stesso tempo, per realizzare l'internalizzazione dei costi ambientali. Per conseguire questi obiettivi ci sono due categorie si strumenti: gli strumenti di regolamentazione diretta, noti come norme di "comando e controllo" e gli strumenti economici puri. In una prima fase, la legislazione in materia di rifiuti ha fatto ampio ricorso agli strumenti tradizionali di command and control, al cui studio ho dedicato il primo capitolo del lavoro. Tuttavia, tali strumenti, per quanto necessari, sono, nella pratica, inefficaci in quanto, come analizzo nel capitolo secondo, sono carenti di adeguati dispositivi di controllo e un apparato sanzionatorio efficace. A causa delle criticità ora elencate, negli anni più recenti si sono diffuse una certa diffidenza e si è perciò affermato il concetto della necessità di avvalersi, accanto agli strumenti di command and control, di strumenti economici puri. In tal senso, già nel 1989 l'Organizzazione per la Cooperazione e lo Sviluppo economico aveva elaborato un documento dal quale si ricavavano almeno quattro categorie di strumenti economici volti alla tutela dell'ambiente, e precisamente: i tributi ambientali, i sussidi, i depositi cauzionali ed i permessi negoziabili. Dal momento che si tratta di una classificazione ancora valida, ho ricostruito gli strumenti economici vigenti, o comunque potenzialmente applicabili, al fine di garantire una corretta gestione dei rifiuti

Expression and selective up-regulation of toxin-related mono ADP-ribosyltransferases by pathogen-associated molecular patterns in alveolar epithelial cells.

Mono ADP-ribosyltransferases (ARTs) are a family of enzymes related to bacterial toxins that possess adenosine diphosphate ribosyltransferase activity. We have assessed that A549 constitutively expressed ART1 on the cell surface and shown that lipotheicoic acid (LTA) and flagellin, but not lipopolysaccharide (LPS), peptidoglycan (PG) and poly (I:C), up-regulate ART1 in a time and dose dependent manner. These agonists did not alter the expression of ART3 and ART5 genes. Indeed, LTA and flagellin stimulation increased the level of ART1 protein and transcript while ART4 gene was activated after stimulation of cells with LPS, LTA, PAM and PG via TLR2 and TLR4 receptors. These results show that human ARTs possess a differential capacity to respond to bacteria cell wall components and might play a crucial role in innate immune response in airway

SMHD: A Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions

Mental health is a significant and growing public health concern. As language usage can be leveraged to obtain crucial insights into mental health conditions, there is a need for large-scale, labeled, mental health-related datasets of users who have been diagnosed with one or more of such conditions. In this paper, we investigate the creation of high-precision patterns to identify self-reported diagnoses of nine different mental health conditions, and obtain high-quality labeled data without the need for manual labelling. We introduce the SMHD (Self-reported Mental Health Diagnoses) dataset and make it available. SMHD is a novel large dataset of social media posts from users with one or multiple mental health conditions along with matched control users. We examine distinctions in users' language, as measured by linguistic and psychological variables. We further explore text classification methods to identify individuals with mental conditions through their language.Comment: COLING 201