Evidence Inference 2.0: More Data, Better Models

How do we most effectively treat a disease or condition? Ideally, we could consult a database of evidence gleaned from clinical trials to answer such questions. Unfortunately, no such database exists; clinical trial results are instead disseminated primarily via lengthy natural language articles. Perusing all such articles would be prohibitively time-consuming for healthcare practitioners; they instead tend to depend on manually compiled systematic reviews of medical literature to inform care. NLP may speed this process up, and eventually facilitate immediate consult of published evidence. The Evidence Inference dataset was recently released to facilitate research toward this end. This task entails inferring the comparative performance of two treatments, with respect to a given outcome, from a particular article (describing a clinical trial) and identifying supporting evidence. For instance: Does this article report that chemotherapy performed better than surgery for five-year survival rates of operable cancers? In this paper, we collect additional annotations to expand the Evidence Inference dataset by 25\%, provide stronger baseline models, systematically inspect the errors that these make, and probe dataset quality. We also release an abstract only (as opposed to full-texts) version of the task for rapid model prototyping. The updated corpus, documentation, and code for new baselines and evaluations are available at http://evidence-inference.ebm-nlp.com/.Comment: Accepted as workshop paper into BioNLP Updated results from SciBERT to Biomed RoBERT

Lay perspectives on hypertension and drug adherence:systematic review of qualitative research

Objective To synthesise the findings from individual qualitative studies on patients’ understanding and experiences of hypertension and drug taking; to investigate whether views differ internationally by culture or ethnic group and whether the research could inform interventions to improve adherence. Design Systematic review and narrative synthesis of qualitative research using the 2006 UK Economic and Social Research Council research methods programme guidance. Data sources Medline, Embase, the British Nursing Index, Social Policy and Practice, and PsycInfo from inception to October 2011. Study selection Qualitative interviews or focus groups among people with uncomplicated hypertension (studies principally in people with diabetes, established cardiovascular disease, or pregnancy related hypertension were excluded). Results 59 papers reporting on 53 qualitative studies were included in the synthesis. These studies came from 16 countries (United States, United Kingdom, Brazil, Sweden, Canada, New Zealand, Denmark, Finland, Ghana, Iran, Israel, Netherlands, South Korea, Spain, Tanzania, and Thailand). A large proportion of participants thought hypertension was principally caused by stress and produced symptoms, particularly headache, dizziness, and sweating. Participants widely intentionally reduced or stopped treatment without consulting their doctor. Participants commonly perceived that their blood pressure improved when symptoms abated or when they were not stressed, and that treatment was not needed at these times. Participants disliked treatment and its side effects and feared addiction. These findings were consistent across countries and ethnic groups. Participants also reported various external factors that prevented adherence, including being unable to find time to take the drugs or to see the doctor; having insufficient money to pay for treatment; the cost of appointments and healthy food; a lack of health insurance; and forgetfulness. Conclusions Non-adherence to hypertension treatment often resulted from patients’ understanding of the causes and effects of hypertension; particularly relying on the presence of stress or symptoms to determine if blood pressure was raised. These beliefs were remarkably similar across ethnic and geographical groups; calls for culturally specific education for individual ethnic groups may therefore not be justified. To improve adherence, clinicians and educational interventions must better understand and engage with patients’ ideas about causality, experiences of symptoms, and concerns about drug side effects

Spá:A Web-Based Viewer for Text Mining in Evidence Based Medicine

Summarizing the evidence about medical interventions is an immense undertaking, in part because unstructured Portable Document Format (PDF) documents remain the main vehicle for disseminating sci- entific findings. Clinicians and researchers must therefore manually ex- tract and synthesise information from these PDFs. We introduce Spá,12 a web-based viewer that enables automated annotation and summari- sation of PDFs via machine learning. To illustrate its functionality, we use Spá to semi-automate the assessment of bias in clinical trials. Spá has a modular architecture, therefore the tool may be widely useful in other domains with a PDF-based literature, including law, physics, and biology

Optimized mobile thin clients through a MPEG-4 BiFS semantic remote display framework

According to the thin client computing principle, the user interface is physically separated from the application logic. In practice only a viewer component is executed on the client device, rendering the display updates received from the distant application server and capturing the user interaction. Existing remote display frameworks are not optimized to encode the complex scenes of modern applications, which are composed of objects with very diverse graphical characteristics. In order to tackle this challenge, we propose to transfer to the client, in addition to the binary encoded objects, semantic information about the characteristics of each object. Through this semantic knowledge, the client is enabled to react autonomously on user input and does not have to wait for the display update from the server. Resulting in a reduction of the interaction latency and a mitigation of the bursty remote display traffic pattern, the presented framework is of particular interest in a wireless context, where the bandwidth is limited and expensive. In this paper, we describe a generic architecture of a semantic remote display framework. Furthermore, we have developed a prototype using the MPEG-4 Binary Format for Scenes to convey the semantic information to the client. We experimentally compare the bandwidth consumption of MPEG-4 BiFS with existing, non-semantic, remote display frameworks. In a text editing scenario, we realize an average reduction of 23% of the data peaks that are observed in remote display protocol traffic

Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization

We consider the problem of automatically generating a narrative biomedical evidence summary from multiple trial reports. We evaluate modern neural models for abstractive summarization of relevant article abstracts from systematic reviews previously conducted by members of the Cochrane collaboration, using the authors conclusions section of the review abstract as our target. We enlist medical professionals to evaluate generated summaries, and we find that modern summarization systems yield consistently fluent and relevant synopses, but that they are not always factual. We propose new approaches that capitalize on domain-specific models to inform summarization, e.g., by explicitly demarcating snippets of inputs that convey key findings, and emphasizing the reports of large and high-quality trials. We find that these strategies modestly improve the factual accuracy of generated summaries. Finally, we propose a new method for automatically evaluating the factuality of generated narrative evidence syntheses using models that infer the directionality of reported findings.Comment: 11 pages, 2 figures. Accepted for presentation at the 2021 AMIA Informatics Summi

Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews

Medical systematic reviews play a vital role in healthcare decision making and policy. However, their production is time-consuming, limiting the availability of high-quality and up-to-date evidence summaries. Recent advancements in large language models (LLMs) offer the potential to automatically generate literature reviews on demand, addressing this issue. However, LLMs sometimes generate inaccurate (and potentially misleading) texts by hallucination or omission. In healthcare, this can make LLMs unusable at best and dangerous at worst. We conducted 16 interviews with international systematic review experts to characterize the perceived utility and risks of LLMs in the specific context of medical evidence reviews. Experts indicated that LLMs can assist in the writing process by drafting summaries, generating templates, distilling information, and crosschecking information. They also raised concerns regarding confidently composed but inaccurate LLM outputs and other potential downstream harms, including decreased accountability and proliferation of low-quality reviews. Informed by this qualitative analysis, we identify criteria for rigorous evaluation of biomedical LLMs aligned with domain expert views.Comment: 18 pages, 2 figures, 8 tables. Accepted as an EMNLP 2023 main pape