Automatic analysis of medical dialogue in the home hemodialysis domain : structure induction and summarization

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2005.Includes bibliographical references (p. 129-134).Spoken medical dialogue is a valuable source of information, and it forms a foundation for diagnosis, prevention and therapeutic management. However, understanding even a perfect transcript of spoken dialogue is challenging for humans because of the lack of structure and the verbosity of dialogues. This work presents a first step towards automatic analysis of spoken medical dialogue. The backbone of our approach is an abstraction of a dialogue into a sequence of semantic categories. This abstraction uncovers structure in informal, verbose conversation between a caregiver and a patient, thereby facilitating automatic processing of dialogue content. Our method induces this structure based on a range of linguistic and contextual features that are integrated in a supervised machine-learning framework. Our model has a classification accuracy of 73%, compared to 33% achieved by a majority baseline (p<0.01). We demonstrate the utility of this structural abstraction by incorporating it into an automatic dialogue summarizer. Our evaluation results indicate that automatically generated summaries exhibit high resemblance to summaries written by humans and significantly outperform random selections (p<0.0001) in precision and recall.(cont.) In addition, task-based evaluation shows that physicians can reasonably answer questions related to patient care by looking at the automatically-generated summaries alone, in contrast to the physicians' performance when they were given summaries from a naive summarizer (p<0.05). This is a significant result because it spares the physician from the need to wade through irrelevant material ample in dialogue transcripts. This work demonstrates the feasibility of automatically structuring and summarizing spoken medical dialogue.by Ronilda Covar Lacson.Ph.D

Assessing the Quality of Annotations in Asthma Gene Expression Experiments

Background: The amount of data deposited in the Gene Expression Omnibus (GEO) has expanded significantly. It is important to ensure that these data are properly annotated with clinical data and descriptions of experimental conditions so that they can be useful for future analysis. This study assesses the adequacy of documented asthma markers in GEO. Three objective measures (coverage, consistency and association) were used for evaluation of annotations contained in 17 asthma studies. Results: There were 918 asthma samples with 20,640 annotated markers. Of these markers, only 10,419 had documented values (50% coverage). In one study carefully examined for consistency, there were discrepancies in drug name usage, with brand name and generic name used in different sections to refer to the same drug. Annotated markers showed adequate association with other relevant variables (i.e. the use of medication only when its corresponding disease state was present). Conclusions: There is inadequate variable coverage within GEO and usage of terms lacks consistency. Association between relevant variables, however, was adequate

Evaluation of a Large-Scale Biomedical Data Annotation Initiative

Background: This study describes a large-scale manual re-annotation of data samples in the Gene Expression Omnibus (GEO), using variables and values derived from the National Cancer Institute thesaurus. A framework is described for creating an annotation scheme for various diseases that is flexible, comprehensive, and scalable. The annotation structure is evaluated by measuring coverage and agreement between annotators. Results: There were 12,500 samples annotated with approximately 30 variables, in each of six disease categories – breast cancer, colon cancer, inflammatory bowel disease (IBD), rheumatoid arthritis (RA), systemic lupus erythematosus (SLE), and Type 1 diabetes mellitus (DM). The annotators provided excellent variable coverage, with known values for over 98% of three critical variables: disease state, tissue, and sample type. There was 89% strict inter-annotator agreement and 92% agreement when using semantic and partial similarity measures. Conclusion: We show that it is possible to perform manual re-annotation of a large repository in a reliable manner

Retrieval of Radiology Reports Citing Critical Findings with Disease-Specific Customization

Background: Communication of critical results from diagnostic procedures between caregivers is a Joint Commission national patient safety goal. Evaluating critical result communication often requires manual analysis of voluminous data, especially when reviewing unstructured textual results of radiologic findings. Information retrieval (IR) tools can facilitate this process by enabling automated retrieval of radiology reports that cite critical imaging findings. However, IR tools that have been developed for one disease or imaging modality often need substantial reconfiguration before they can be utilized for another disease entity. Purpose: This paper: 1) describes the process of customizing two Natural Language Processing (NLP) and Information Retrieval/Extraction applications – an open-source toolkit, A Nearly New Information Extraction system (ANNIE); and an application developed in-house, Information for Searching Content with an Ontology-Utilizing Toolkit (iSCOUT) – to illustrate the varying levels of customization required for different disease entities and; 2) evaluates each application’s performance in identifying and retrieving radiology reports citing critical imaging findings for three distinct diseases, pulmonary nodule, pneumothorax, and pulmonary embolus. Results: Both applications can be utilized for retrieval. iSCOUT and ANNIE had precision values between 0.90-0.98 and recall values between 0.79 and 0.94. ANNIE had consistently higher precision but required more customization. Conclusion: Understanding the customizations involved in utilizing NLP applications for various diseases will enable users to select the most suitable tool for specific tasks

Impact of an Information Technology–Enabled Initiative on the Quality of Prostate Multiparametric MRI Reports

Rationale and Objectives: Assess the impact of implementing a structured report template and a computer-aided diagnosis (CAD) tool on the quality of prostate multiparametric MRI (mp-MRI) reports. Materials and Methods: Institutional Review Board approval was obtained for this HIPAA-compliant study performed at an academic medical center. The study cohort included all prostate mp-MRI reports (n=385) finalized 6 months before and after implementation of a structured report template and a CAD tool (collectively the IT tools) integrated into the PACS workstation. Primary outcome measure was quality of prostate mp-MRI reports. An expert panel of our institution’s subspecialty trained abdominal radiologists defined prostate mp-MRI report quality as optimal, satisfactory or unsatisfactory based on documentation of 9 variables. Reports were reviewed to extract the predefined quality variables and determine whether the IT tools were used to create each report. Chi-square and Student’s t-tests were used to compare report quality before and after implementation of IT tools. Results: The overall proportion of optimal or satisfactory reports increased from 29.8% (47/158) to 53.3% (121/227) (p<0.001) after implementing the IT tools. While the proportion of optimal or satisfactory reports increased among reports generated using at least one of the IT tools (47/158=[29.8%] vs. 105/161=[65.2%]; p<0.001), there was no change in quality among reports generated without use of the IT tools (47/158=[29.8%] vs. 16/66=[24.2%]; p=0.404). Conclusion: The use of a structured template and CAD tool improved the quality of prostate mp-MRI reports compared to free-text report format and subjective measurement of contrast enhancement kinetic curve

Assessing Strength of Evidence of Appropriate Use Criteria for Diagnostic Imaging Examinations

Objective For health information technology tools to fully inform evidence-based decisions, recommendations must be reliably assessed for quality and strength of evidence. We aimed to create an annotation framework for grading recommendations regarding appropriate use of diagnostic imaging examinations. Methods The annotation framework was created by an expert panel (clinicians in three medical specialties, medical librarians, and biomedical scientists) who developed a process for achieving consensus in assessing recommendations, and evaluated by measuring agreement in grading the strength of evidence for 120 empirically selected recommendations using the Oxford Levels of Evidence. Results Eighty-two percent of recommendations were assigned to Level 5 (expert opinion). Inter-annotator agreement was 0.70 on initial grading (κ = 0.35, 95% CI, 0.23-0.48). After systematic discussion utilizing the annotation framework, agreement increased significantly to 0.97 (κ = 0.88, 95% CI, 0.77-0.99). Conclusions A novel annotation framework was effective for grading the strength of evidence supporting appropriate use criteria for diagnostic imaging exams