Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understanding

Large language models (LLMs) have made significant progress in various domains, including healthcare. However, the specialized nature of clinical language understanding tasks presents unique challenges and limitations that warrant further investigation. In this study, we conduct a comprehensive evaluation of state-of-the-art LLMs, namely GPT-3.5, GPT-4, and Bard, within the realm of clinical language understanding tasks. These tasks span a diverse range, including named entity recognition, relation extraction, natural language inference, semantic textual similarity, document classification, and question-answering. We also introduce a novel prompting strategy, self-questioning prompting (SQP), tailored to enhance LLMs' performance by eliciting informative questions and answers pertinent to the clinical scenarios at hand. Our evaluation underscores the significance of task-specific learning strategies and prompting techniques for improving LLMs' effectiveness in healthcare-related tasks. Additionally, our in-depth error analysis on the challenging relation extraction task offers valuable insights into error distribution and potential avenues for improvement using SQP. Our study sheds light on the practical implications of employing LLMs in the specialized domain of healthcare, serving as a foundation for future research and the development of potential applications in healthcare settings.Comment: 19 pages, preprin

Towards Interoperability in E-health Systems: a three-dimensional approach based on standards and semantics

Proceedings of: HEALTHINF 2009 (International Conference on Helath Informatics), Porto (Portugal), January 14-17, 2009, is part of BIOSTEC (Intemational Joint Conference on Biomedical Engineering Systems and Technologies)The interoperability problem in eHealth can only be addressed by mean of combining standards and technology. However, these alone do not suffice. An appropiate framework that articulates such combination is required. In this paper, we adopt a three-dimensional (information, conference and inference) approach for such framework, based on OWL as formal language for terminological and ontological health resources, SNOMED CT as lexical backbone for all such resources, and the standard CEN 13606 for representing EHRs. Based on tha framewok, we propose a novel form for creating and supporting networks of clinical terminologies. Additionally, we propose a number of software modules to semantically process and exploit EHRs, including NLP-based search and inference, wich can support medical applications in heterogeneous and distributed eHealth systems.This work has been funded as part of the Spanish nationally funded projects ISSE (FIT-350300-2007-75) and CISEP (FIT-350301-2007-18). We also acknowledge IST-2005-027595 EU project NeO