BI-LAVA: Biocuration with Hierarchical Image Labeling through Active Learning and Visual Analysis

In the biomedical domain, taxonomies organize the acquisition modalities of scientific images in hierarchical structures. Such taxonomies leverage large sets of correct image labels and provide essential information about the importance of a scientific publication, which could then be used in biocuration tasks. However, the hierarchical nature of the labels, the overhead of processing images, the absence or incompleteness of labeled data, and the expertise required to label this type of data impede the creation of useful datasets for biocuration. From a multi-year collaboration with biocurators and text-mining researchers, we derive an iterative visual analytics and active learning strategy to address these challenges. We implement this strategy in a system called BI-LAVA Biocuration with Hierarchical Image Labeling through Active Learning and Visual Analysis. BI-LAVA leverages a small set of image labels, a hierarchical set of image classifiers, and active learning to help model builders deal with incomplete ground-truth labels, target a hierarchical taxonomy of image modalities, and classify a large pool of unlabeled images. BI-LAVA's front end uses custom encodings to represent data distributions, taxonomies, image projections, and neighborhoods of image thumbnails, which help model builders explore an unfamiliar image dataset and taxonomy and correct and generate labels. An evaluation with machine learning practitioners shows that our mixed human-machine approach successfully supports domain experts in understanding the characteristics of classes within the taxonomy, as well as validating and improving data quality in labeled and unlabeled collections.Comment: 15 pages, 6 figure

Roses Have Thorns: Understanding the Downside of Oncological Care Delivery Through Visual Analytics and Sequential Rule Mining

Personalized head and neck cancer therapeutics have greatly improved survival rates for patients, but are often leading to understudied long-lasting symptoms which affect quality of life. Sequential rule mining (SRM) is a promising unsupervised machine learning method for predicting longitudinal patterns in temporal data which, however, can output many repetitive patterns that are difficult to interpret without the assistance of visual analytics. We present a data-driven, human-machine analysis visual system developed in collaboration with SRM model builders in cancer symptom research, which facilitates mechanistic knowledge discovery in large scale, multivariate cohort symptom data. Our system supports multivariate predictive modeling of post-treatment symptoms based on during-treatment symptoms. It supports this goal through an SRM, clustering, and aggregation back end, and a custom front end to help develop and tune the predictive models. The system also explains the resulting predictions in the context of therapeutic decisions typical in personalized care delivery. We evaluate the resulting models and system with an interdisciplinary group of modelers and head and neck oncology researchers. The results demonstrate that our system effectively supports clinical and symptom research

FixingTIM: Interactive exploration of sequence and structural data to identify functional mutations in protein families

Background: Knowledge of the 3D structure and functionality of proteins can lead to insight into the associated cellular processes, speed up the creation of pharmaceutical products, and develop drugs that are more effective in combating disease. Methods: We present the design and implementation of a visual mining and analysis tool to help identify protein mutations across a family of structural models and to help discover the effect of these mutations on protein function. We integrate 3D structure and sequence information in a common visual interface; multiple linked views and a computational backbone allow comparison at the molecular and atomic levels, while a novel trend-image visual abstraction allows for the sorting and mining of large collections of sequences and of their residues. Results: We evaluate our approach on the triosephosphate isomerase (TIM) family structural models and sequence data and show that our tool provides an effective, scalable way to navigate a family of proteins, as well as a means to inspect the structure and sequence of individual proteins. Conclusions: The TIM application shows that our tool can assist in the navigation of families of proteins, as well as in the exploration of individual protein structures. In conjunction with domain expert knowledge, this interactive tool can help provide biophysical insight into why specific mutations affect function and potentially suggest additional modifications to the protein that could be used to rescue functionality

DASS Good: Explainable Data Mining of Spatial Cohort Data

Developing applicable clinical machine learning models is a difficult task when the data includes spatial information, for example, radiation dose distributions across adjacent organs at risk. We describe the co-design of a modeling system, DASS, to support the hybrid human-machine development and validation of predictive models for estimating long-term toxicities related to radiotherapy doses in head and neck cancer patients. Developed in collaboration with domain experts in oncology and data mining, DASS incorporates human-in-the-loop visual steering, spatial data, and explainable AI to augment domain knowledge with automatic data mining. We demonstrate DASS with the development of two practical clinical stratification models and report feedback from domain experts. Finally, we describe the design lessons learned from this collaborative experience.Comment: 10 pages, 9 figure

Utilizing image and caption information for biomedical document classification.

MOTIVATION: Biomedical research findings are typically disseminated through publications. To simplify access to domain-specific knowledge while supporting the research community, several biomedical databases devote significant effort to manual curation of the literature-a labor intensive process. The first step toward biocuration requires identifying articles relevant to the specific area on which the database focuses. Thus, automatically identifying publications relevant to a specific topic within a large volume of publications is an important task toward expediting the biocuration process and, in turn, biomedical research. Current methods focus on textual contents, typically extracted from the title-and-abstract. Notably, images and captions are often used in publications to convey pivotal evidence about processes, experiments and results. RESULTS: We present a new document classification scheme, using both image and caption information, in addition to titles-and-abstracts. To use the image information, we introduce a new image representation, namely Figure-word, based on class labels of subfigures. We use word embeddings for representing captions and titles-and-abstracts. To utilize all three types of information, we introduce two information integration methods. The first combines Figure-words and textual features obtained from captions and titles-and-abstracts into a single larger vector for document representation; the second employs a meta-classification scheme. Our experiments and results demonstrate the usefulness of the newly proposed Figure-words for representing images. Moreover, the results showcase the value of Figure-words, captions and titles-and-abstracts in providing complementary information for document classification; these three sources of information when combined, lead to an overall improved classification performance. AVAILABILITY AND IMPLEMENTATION: Source code and the list of PMIDs of the publications in our datasets are available upon request

Multi-organ spatial stratification of 3-D dose distributions improves risk prediction of long-term self-reported severe symptoms in oropharyngeal cancer patients receiving radiotherapy:development of a pre-treatment decision support tool

PURPOSE: Identify Oropharyngeal cancer (OPC) patients at high-risk of developing long-term severe radiation-associated symptoms using dose volume histograms for organs-at-risk, via unsupervised clustering.MATERIAL AND METHODS: All patients were treated using radiation therapy for OPC. Dose-volume histograms of organs-at-risk were extracted from patients' treatment plans. Symptom ratings were collected via the MD Anderson Symptom Inventory (MDASI) given weekly during, and 6 months post-treatment. Drymouth, trouble swallowing, mucus, and vocal dysfunction were selected for analysis in this study. Patient stratifications were obtained by applying Bayesian Mixture Models with three components to patient's dose histograms for relevant organs. The clusters with the highest total mean doses were translated into dose thresholds using rule mining. Patient stratifications were compared against Tumor staging information using multivariate likelihood ratio tests. Model performance for prediction of moderate/severe symptoms at 6 months was compared against normal tissue complication probability (NTCP) models using cross-validation.RESULTS: A total of 349 patients were included for long-term symptom prediction. High-risk clusters were significantly correlated with outcomes for severe late drymouth (p &lt;.0001, OR = 2.94), swallow (p = .002, OR = 5.13), mucus (p = .001, OR = 3.18), and voice (p = .009, OR = 8.99). Simplified clusters were also correlated with late severe symptoms for drymouth (p &lt;.001, OR = 2.77), swallow (p = .01, OR = 3.63), mucus (p = .01, OR = 2.37), and voice (p &lt;.001, OR = 19.75). Proposed cluster stratifications show better performance than NTCP models for severe drymouth (AUC.598 vs.559, MCC.143 vs.062), swallow (AUC.631 vs.561, MCC.20 vs -.030), mucus (AUC.596 vs.492, MCC.164 vs -.041), and voice (AUC.681 vs.555, MCC.181 vs -.019). Simplified dose thresholds also show better performance than baseline models for predicting late severe ratings for all symptoms.CONCLUSION: Our results show that leveraging the 3-D dose histograms from radiation therapy plan improves stratification of patients according to their risk of experiencing long-term severe radiation associated symptoms, beyond existing NTPC models. Our rule-based method can approximate our stratifications with minimal loss of accuracy and can proactively identify risk factors for radiation-associated toxicity.</p

Head and neck cancer predictive risk estimator to determine control and therapeutic outcomes of radiotherapy (HNC-PREDICTOR):development, international multi-institutional validation, and web implementation of clinic-ready model-based risk stratification for head and neck cancer

Background: Personalised radiotherapy can improve treatment outcomes of patients with head and neck cancer (HNC), where currently a ‘one-dose-fits-all’ approach is the standard. The aim was to establish individualised outcome prediction based on multi-institutional international ‘big-data’ to facilitate risk-based stratification of patients with HNC. Methods: The data of 4611 HNC radiotherapy patients from three academic cancer centres were split into four cohorts: a training (n = 2241), independent test (n = 786), and external validation cohorts 1 (n = 1087) and 2 (n = 497). Tumour- and patient-related clinical variables were considered in a machine learning pipeline to predict overall survival (primary end-point) and local and regional tumour control (secondary end-points); serially, imaging features were considered for optional model improvement. Finally, patients were stratified into high-, intermediate-, and low-risk groups. Results: Performance score, AJCC8th stage, pack-years, and Age were identified as predictors for overall survival, demonstrating good performance in both the training cohort (c-index = 0.72 [95% CI, 0.66–0.77]) and in all three validation cohorts (c-indices: 0.76 [0.69–0.83], 0.73 [0.68–0.77], and 0.75 [0.68–0.80]). Excellent stratification of patients with HNC into high, intermediate, and low mortality risk was achieved; with 5-year overall survival rates of 17–46% for the high-risk group compared to 92–98% for the low-risk group. The addition of morphological image feature further improved the performance (c-index = 0.73 [0.64–0.81]). These models are integrated in a clinic-ready interactive web interface: https://uic-evl.github.io/hnc-predictor/ Conclusions: Robust model-based prediction was able to stratify patients with HNC in distinct high, intermediate, and low mortality risk groups. This can effectively be capitalised for personalised radiotherapy, e.g., for tumour radiation dose escalation/de-escalation