Event knowledge in large language models: the gap between the impossible and the unlikely

Word co-occurrence patterns in language corpora contain a surprising amount of conceptual knowledge. Large language models (LLMs), trained to predict words in context, leverage these patterns to achieve impressive performance on diverse semantic tasks requiring world knowledge. An important but understudied question about LLMs' semantic abilities is whether they acquire generalized knowledge of common events. Here, we test whether five pre-trained LLMs (from 2018's BERT to 2023's MPT) assign higher likelihood to plausible descriptions of agent-patient interactions than to minimally different implausible versions of the same event. Using three curated sets of minimal sentence pairs (total n=1,215), we found that pre-trained LLMs possess substantial event knowledge, outperforming other distributional language models. In particular, they almost always assign higher likelihood to possible vs. impossible events (The teacher bought the laptop vs. The laptop bought the teacher). However, LLMs show less consistent preferences for likely vs. unlikely events (The nanny tutored the boy vs. The boy tutored the nanny). In follow-up analyses, we show that (i) LLM scores are driven by both plausibility and surface-level sentence features, (ii) LLM scores generalize well across syntactic variants (active vs. passive constructions) but less well across semantic variants (synonymous sentences), (iii) some LLM errors mirror human judgment ambiguity, and (iv) sentence plausibility serves as an organizing dimension in internal LLM representations. Overall, our results show that important aspects of event knowledge naturally emerge from distributional linguistic patterns, but also highlight a gap between representations of possible/impossible and likely/unlikely events.Comment: The two lead authors have contributed equally to this wor

PW06-05 The predictive role of anxiety disorders on depressive phenomenology during post-partum period

Aims:To investigate the predictive role of any specific (DSM-IV) Anxiety Disorders (AD) on depressive symptoms and Major or Minor Depressive Disorder (MDD, mDD) during early postpartum period.Method:Women (at the 12th-15th gestational week, N=1066) were recruited in the framework of the Program 'Perinatal Depression - Research & Screening Unit (PND-ReScU)". Depressive symptoms were assessed by the Edinburgh Postnatal Depression Scale (EPDS), and Axis-I disorders (AD, MDD, mDD) were diagnosed with the Structured Clinical Interview for Axis-I Disorders (SCID-I).Results:Any current AD at baseline (3rd month of pregnancy) was detected in 231 (21.7%). Having at least one current AD, was associated with a greater likelihood of having MDD or mDD during the early postpartum period, even after the adjustment for the confounding factor of having a lifetime history of MDD (RR=3.86 95%CI 1.58-9.42).In particular, women affected by Obsessive Compulsive Disorder (N=17; 1.6%) or Panic Disorder (N=43; 4%) had at higher risk to develop depressive symptoms (EPDS≥13) during the postpartum period than women without these AD (RR=6.9 and 6.7 respectively). As for the risk of developing PPD, the strongest association was found for women with Panic Disorder (RR=7.6 95% CI 2.62-22.0).Conclusions:AD are associated with a greater likelihood to develop depressive symptoms and MDD or mDD during the early postpartum period. Women with current PD have the strongest risk to develop both MDD or mDD and depressive symptoms during early postpartum period compared to other anxiety disorders

EVALITA Evaluation of NLP and Speech Tools for Italian - December 17th, 2020

Welcome to EVALITA 2020! EVALITA is the evaluation campaign of Natural Language Processing and Speech Tools for Italian. EVALITA is an initiative of the Italian Association for Computational Linguistics (AILC, http://www.ai-lc.it) and it is endorsed by the Italian Association for Artificial Intelligence (AIxIA, http://www.aixia.it) and the Italian Association for Speech Sciences (AISV, http://www.aisv.it)