Extrinsically-Focused Evaluation of Omissions in Medical Summarization

The goal of automated summarization techniques (Paice, 1990; Kupiec et al, 1995) is to condense text by focusing on the most critical information. Generative large language models (LLMs) have shown to be robust summarizers, yet traditional metrics struggle to capture resulting performance (Goyal et al, 2022) in more powerful LLMs. In safety-critical domains such as medicine, more rigorous evaluation is required, especially given the potential for LLMs to omit important information in the resulting summary. We propose MED-OMIT, a new omission benchmark for medical summarization. Given a doctor-patient conversation and a generated summary, MED-OMIT categorizes the chat into a set of facts and identifies which are omitted from the summary. We further propose to determine fact importance by simulating the impact of each fact on a downstream clinical task: differential diagnosis (DDx) generation. MED-OMIT leverages LLM prompt-based approaches which categorize the importance of facts and cluster them as supporting or negating evidence to the diagnosis. We evaluate MED-OMIT on a publicly-released dataset of patient-doctor conversations and find that MED-OMIT captures omissions better than alternative metrics

Algorithms for automated diagnosis of cardiovascular diseases based on ECG data: A comprehensive systematic review

The prevalence of cardiovascular diseases is increasing around the world. However, the technology is evolving and can be monitored with low-cost sensors anywhere at any time. This subject is being researched, and different methods can automatically identify these diseases, helping patients and healthcare professionals with the treatments. This paper presents a systematic review of disease identification, classification, and recognition with ECG sensors. The review was focused on studies published between 2017 and 2022 in different scientific databases, including PubMed Central, Springer, Elsevier, Multidisciplinary Digital Publishing Institute (MDPI), IEEE Xplore, and Frontiers. It results in the quantitative and qualitative analysis of 103 scientific papers. The study demonstrated that different datasets are available online with data related to various diseases. Several ML/DP-based models were identified in the research, where Convolutional Neural Network and Support Vector Machine were the most applied algorithms. This review can allow us to identify the techniques that can be used in a system that promotes the patient’s autonomy.N/

Evaluation of the Clinical Integration Model for Hospital Care Delivery

Purpose: Two studies were used to evaluate whether introduction of the Clinical Integration Model (CIM) would decrease cost, length of stay (LOS), and mortality in two populations: a psychiatric in-patient population and congestive heart failure (CHF) patients. Objectives: 1. Evaluate reliability and validity of a process tool, the CareGraph, essential in the CIM. 2. Determine if there is a difference for LOS and cost between patients receiving care in the CIM and those receiving care in a traditional primary care delivery model in a psychiatric population; compare the same parameters as well as survival in the CHF population. Methods: Reliability of the CareGraph tool was evaluated using Cronbach™s alpha, and known-groups validity was evaluated using a t-test to compare admission and discharge scores. A retrospective pre-implementation, post-implementation design was utilized to evaluate outcomes in the psychiatric population. A retrospective comparative design was used in the CHF population. Results: Initial Cronbach™ alpha for all CareGraph items was .71. For the psychiatric population, LOS increased between 2010 (4 days) and 2011 (5 days) (t [189] = -2.71, p\u3c.01). Although the LOS was longer after implementation of the CIM, the cost was not significantly different. Evaluation of differences between CIM hospitals and regular care hospitals using the inpatient CHF population showed a significant difference in two outcome variables; LOS, F(3, 245) = 5.78, p = .001 and cost F(3,226) = 21.70, p = .000 but no difference in survival rates

Remote Patient Monitoring: Decrease Rehospitalization for Spinal Cord Injury Patients

Practice Problem: The lengthy distance required to access specialty care, the overall higher cost of SCI/D care, complications associated with SCI, and the potential negative impact of shortened hospital stays are all compelling reasons to use telehealth technologies to deliver specialty services for medical issues. PICOT: The PICOT question that guided this project was in adult spinal cord injury patients with chronic disease receiving primary care at a spinal cord injury center (P), how does the implementation of a remote patient monitoring home telehealth for SCI patients recently discharged from acute-care setting (I) compared to the usual practice of one post-discharge follow up phone call at 7 days (C), improve early recognition of patient deterioration to prevent acute care rehospitalization (O) within 30 days of discharge (T). Evidence: Spinal Cord injury patients are at risk for developing complications after injury. Paststudies have demonstrated the effectiveness of telehealth to prevent rehospitalization, which suggests the potential of telehealth on post-discharge follow-up care. Intervention: Implement remote patient monitoring home telehealth for SCI patients meeting the criteria for high-risk rehospitalization. Outcome: The pilot project results have a positive correlation with the reduction of 30-day hospital readmission rates for SCI patients participating in the RPM. During the pilot period, no readmissions occurred for the RPM participants, whereas those who declined participation were readmitted at a rate of 22%. Clinical significant findings of improved outcomes and reduced 30-day readmissions are supported through this pilot project. Conclusion: The project utilized the Johns Hopkins evidence-based model’s three-step PET framework and Roger’s diffusion of innovation change theory to support reduced rehospitalization for SCI patients through RPM

Development of a heart failure filter for Medline: an objective approach using evidence-based clinical practice guidelines as an alternative to hand searching

<p>Abstract</p> <p>Background</p> <p>Heart failure is a highly debilitating syndrome with a poor prognosis primarily affecting the elderly. Clinicians wanting timely access to heart failure evidence to provide optimal patient care can face many challenges in locating this evidence.</p> <p>This study developed and validated a search filter of high clinical utility for the retrieval of heart failure articles in OvidSP Medline.</p> <p>Methods</p> <p>A Clinical Advisory Group was established to advise study investigators. The study set of 876 relevant articles from four heart failure clinical practice guidelines was divided into three datasets: a Term Identification Set, a Filter Development Set, and a Filter Validation Set. A further validation set (the Cochrane Validation Set) was formed using studies included in Cochrane heart failure systematic reviews. Candidate search terms were identified via word frequency analysis. The filter was developed by creating combinations of terms and recording their performance in retrieving items from the Filter Development Set. The filter's recall was then validated in both the Filter Validation Set and the Cochrane Validation Set. A precision estimate was obtained post-hoc by running the filter in Medline and screening the first 200 retrievals for relevance to heart failure.</p> <p>Results</p> <p>The four-term filter achieved a recall of 96.9% in the Filter Development Set; 98.2% in the Filter Validation Set; and 97.8% in the Cochrane Validation Set. Of the first 200 references retrieved by the filter when run in Medline, 150 were deemed relevant and 50 irrelevant. The post-hoc precision estimate was therefore 75%.</p> <p>Conclusions</p> <p>This study describes an objective method for developing a validated heart failure filter of high recall performance and then testing its precision post-hoc. Clinical practice guidelines were found to be a feasible alternative to hand searching in creating a gold standard for filter development. Guidelines may be especially appropriate given their clinical utility. A validated heart failure filter is now available to support health professionals seeking reliable and efficient access to the heart failure literature.</p

Doctor of Philosophy

DissertationHealth information technology (HIT) in conjunction with quality improvement (QI) methodologies can promote higher quality care at lower costs. Unfortunately, most inpatient hospital settings have been slow to adopt HIT and QI methodologies. Successful adoption requires close attention to workflow. Workflow is the sequence of tasks, processes, and the set of people or resources needed for those tasks that are necessary to accomplish a given goal. Assessing the impact on workflow is an important component of determining whether a HIT implementation will be successful, but little research has been conducted on the impact of eMeasure (electronic performance measure) implementation on workflow. One solution to addressing implementation challenges such as the lack of attention to workflow is an implementation toolkit. An implementation toolkit is an assembly of instruments such as checklists, forms, and planning documents. We developed an initial eMeasure Implementation Toolkit for the heart failure (HF) eMeasure to allow QI and information technology (IT) professionals and their team to assess the impact of implementation on workflow. During the development phase of the toolkit, we undertook a literature review to determine the components of the toolkit. We conducted stakeholder interviews with HIT and QI key informants and subject matter experts (SMEs) at the US Department of Veteran Affairs (VA). Key informants provided a broad understanding about the context of workflow during eMeasure implementation. Based on snowball sampling, we also interviewed other SMEs based on the recommendations of the key informants who suggested tools and provided information essential to the toolkit development. The second phase involved evaluation of the toolkit for relevance and clarity, by experts in non-VA settings. The experts evaluated the sections of the toolkit that contained the tools, via a survey. The final toolkit provides a distinct set of resources and tools, which were iteratively developed during the research and available to users in a single source document. The research methodology provided a strong unified overarching implementation framework in the form of the Promoting Action on Research Implementation in Health Services (PARIHS) model in combination with a sociotechnical model of HIT that strengthened the overall design of the study