Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations

Biomedical literature is growing rapidly, making it challenging to curate and extract knowledge manually. Biomedical natural language processing (BioNLP) techniques that can automatically extract information from biomedical literature help alleviate this burden. Recently, large Language Models (LLMs), such as GPT-3 and GPT-4, have gained significant attention for their impressive performance. However, their effectiveness in BioNLP tasks and impact on method development and downstream users remain understudied. This pilot study (1) establishes the baseline performance of GPT-3 and GPT-4 at both zero-shot and one-shot settings in eight BioNLP datasets across four applications: named entity recognition, relation extraction, multi-label document classification, and semantic similarity and reasoning, (2) examines the errors produced by the LLMs and categorized the errors into three types: missingness, inconsistencies, and unwanted artificial content, and (3) provides suggestions for using LLMs in BioNLP applications. We make the datasets, baselines, and results publicly available to the community via https://github.com/qingyu-qc/gpt_bionlp_benchmark

Methods for extending biomedical reference ontologies and interface terminologies for EHRr text annotation

Biomedical ontologies and terminologies are a cornerstone in various electronic health record systems (EHRs) for encoding information related to diseases, diagnoses, treatments, etc. Ontologies in general represent entities (concepts) and events along with all interdependent properties and relationships in an efficient way to facilitate easy access, retrieval and sharing. With the landscape of medicine rapidly changing, biomedical ontologies and terminologies need to rapidly evolve to support interoperability, medical coding, record keeping, and healthcare activities in general, and to facilitate interdisciplinary research. Extending ontologies by identifying new and missing concepts plays a vital role in the maintenance of ontologies to keep up with the constant changes. Even though different biomedical ontologies capture knowledge in a wide variety of medical domains, they still have substantial overlap in their conceptual content. This dissertation explores various methodologies that can be used to enrich the content of biomedical ontologies and terminologies. The dissertation is divided into two parts. The first part addresses how cross-ontology topological patterns can be designed and used to identify missing concepts. The methods presented involve comparing horizontal and vertical density differences between identical concepts in two ontologies. Horizontal density studies identified cases of missing child concepts, alternative classifications, synonyms, and errors in ontologies. A deeper analysis of alternative classification is performed. These alternative classifications are analyzed, and a metric is presented for identifying likely cases of such alternative classifications. Vertical density differences occur when a concept is missing on a path in one ontology but exists in the other one. Furthermore, topological patterns involving three terminologies are presented. A pattern named fire ladder incorporates both vertical and horizontal density differences among three terminologies supporting concepts import. Biomedical ontologies are developed with great investment of time, effort, and budget. Are biomedical ontologies regularly maintained? If not, what are the root causes behind this? A detailed investigation of these questions is conducted both from a quantitative and qualitative perspective. Ontologies and terminologies are not the only sources of medical concepts. Large repositories of unstructured medical text exist in EHRs. Preliminary studies reveal that reference ontologies and terminologies do not contain many of the frequently recorded fine granularity concepts in EHRs. Recently, with the COVID-19 pandemic, EHRs have been accumulating information regarding new symptoms, procedures and tests that are not all currently present in existing reference ontologies and terminologies. To overcome these issues, in the second part of the dissertation, natural language processing techniques to mine concepts from clinical text are presented. The mined concepts are incorporated into interface terminologies that are catering to the annotation of EHR text in different medical specialties. Mining clinical text to create a COVID interface terminology and a Cardiology interface terminology are discussed. EHR annotation enables secondary use of EHR text for clinical research, such as identifying eligible patients for clinical trials