Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report

Purpose: Large Language Models (LLMs) hold significant promise for medical applications. Retrieval Augmented Generation (RAG) emerges as a promising approach for customizing domain knowledge in LLMs. This case study presents the development and evaluation of an LLM-RAG pipeline tailored for healthcare, focusing specifically on preoperative medicine. Methods: We developed an LLM-RAG model using 35 preoperative guidelines and tested it against human-generated responses, with a total of 1260 responses evaluated. The RAG process involved converting clinical documents into text using Python-based frameworks like LangChain and Llamaindex, and processing these texts into chunks for embedding and retrieval. Vector storage techniques and selected embedding models to optimize data retrieval, using Pinecone for vector storage with a dimensionality of 1536 and cosine similarity for loss metrics. Human-generated answers, provided by junior doctors, were used as a comparison. Results: The LLM-RAG model generated answers within an average of 15-20 seconds, significantly faster than the 10 minutes typically required by humans. Among the basic LLMs, GPT4.0 exhibited the best accuracy of 80.1%. This accuracy was further increased to 91.4% when the model was enhanced with RAG. Compared to the human-generated instructions, which had an accuracy of 86.3%, the performance of the GPT4.0 RAG model demonstrated non-inferiority (p=0.610). Conclusions: In this case study, we demonstrated a LLM-RAG model for healthcare implementation. The pipeline shows the advantages of grounded knowledge, upgradability, and scalability as important aspects of healthcare LLM deployment.Comment: N

Burnout, anxiety and depression in healthcare workers during the early COVID-19 period in Singapore

acceptedVersionPeer reviewe

National survey of outcomes and practices in acute respiratory distress syndrome in Singapore

The authors acknowledge the following as the total funding sources for this study: 1. SICM NICER grant: logistical, non-monetary, support from the Society of Intensive Care Medicine Singapore. This was in the form of Ngee Ann Polytechnic students (8) who collected the data for the study for one month. 2. NMRC (National medical research council) grant for Dr, Matthew Cove (partial support for this study): This was in the shape of salary support for all his research related activity. (NMRC/TA/0015/2013) (MEC)

Assessment of acute kidney injury risk using a machine-learning guided generalized structural equation model: a cohort study

10.1186/s12882-021-02238-9BMC Nephrology2216

Healthcare worker stress, anxiety and burnout during the COVID-19 pandemic in Singapore : A 6-month multi-centre prospective study

Aim The long-term stress, anxiety and job burnout experienced by healthcare workers (HCWs) are important to consider as the novel coronavirus disease (COVID-19) pandemic stresses healthcare systems globally. The primary objective was to examine the changes in the proportion of HCWs reporting stress, anxiety, and job burnout over six months during the peak of the pandemic in Singapore. The secondary objective was to examine the extent that objective job characteristics, HCW-perceived job factors, and HCW personal resources were associated with stress, anxiety, and job burnout. Method A sample of HCWs (doctors, nurses, allied health professionals, administrative and operations staff; N = 2744) was recruited via invitation to participate in an online survey from four tertiary hospitals. Data were gathered between March-August 2020, which included a 2-month lockdown period. HCWs completed monthly web-based self-reported assessments of stress (Perceived Stress Scale-4), anxiety (Generalized Anxiety Disorder-7), and job burnout (Physician Work Life Scale). Results The majority of the sample consisted of female HCWs (81%) and nurses (60%). Using random-intercept logistic regression models, elevated perceived stress, anxiety and job burnout were reported by 33%, 13%, and 24% of the overall sample at baseline respectively. The proportion of HCWs reporting stress and job burnout increased by approximately 1·0% and 1·2% respectively per month. Anxiety did not significantly increase. Working long hours was associated with higher odds, while teamwork and feeling appreciated at work were associated with lower odds, of stress, anxiety, and job burnout. Conclusions Perceived stress and job burnout showed a mild increase over six months, even after exiting the lockdown. Teamwork and feeling appreciated at work were protective and are targets for developing organizational interventions to mitigate expected poor outcomes among frontline HCWs.publishedVersionPeer reviewe