Combined information from Raman spectroscopy and optical coherence tomography for enhanced diagnostic accuracy in tissue discrimination

We thank the UK EPSRC for funding, the CR-UK/EPSRC/MRC/DoH (England) imaging programme, the European Union project FAMOS (FP7 ICT, contract no. 317744) and the European Union project IIIOS (FP7/2007-2013, contract no. 238802). We thank Tayside Tissue Bank for providing us with the tissue samples under request number TR000289. K.D. is a Royal Society-Wolfson Merit Award Holder.Optical spectroscopy and imaging methods have proved to have potential to discriminate between normal and abnormal tissue types through minimally invasive procedures. Raman spectroscopy and Optical Coherence Tomography (OCT) provides chemical and morphological information of tissues respectively, which are complementary to each other. When used individually they might not be able to obtain high enough sensitivity and specificity that is clinically relevant. In this study we combined Raman spectroscopy information with information obtained from OCT to enhance the sensitivity and specificity in discriminating between Colonic Adenocarcinoma from Normal Colon. OCT being an imaging technique, the information from this technique is conventionally analyzed qualitatively. To combine with Raman spectroscopy information, it was essential to quantify the morphological information obtained from OCT. Texture analysis was used to extract information from OCT images, which in-turn was combined with the information obtained from Raman spectroscopy. The sensitivity and specificity of the classifier was estimated using leave one out cross validation (LOOCV) method where support vector machine (SVM) was used for binary classification of the tissues. The sensitivity obtained using Raman spectroscopy and OCT individually was 89% and 78% respectively and the specificity was 77% and 74% respectively. Combining the information derived using the two techniques increased both sensitivity and specificity to 94% demonstrating that combining complementary optical information enhances diagnostic accuracy. These results demonstrate that a multimodal approach using Raman-OCT would be able to enhance the diagnostic accuracy for identifying normal and cancerous tissue types.Publisher PD

MELINDA: A Multimodal Dataset for Biomedical Experiment Method Classification

We introduce a new dataset, MELINDA, for Multimodal biomEdicaL experImeNt methoD clAssification. The dataset is collected in a fully automated distant supervision manner, where the labels are obtained from an existing curated database, and the actual contents are extracted from papers associated with each of the records in the database. We benchmark various state-of-the-art NLP and computer vision models, including unimodal models which only take either caption texts or images as inputs, and multimodal models. Extensive experiments and analysis show that multimodal models, despite outperforming unimodal ones, still need improvements especially on a less-supervised way of grounding visual concepts with languages, and better transferability to low resource domains. We release our dataset and the benchmarks to facilitate future research in multimodal learning, especially to motivate targeted improvements for applications in scientific domains.Comment: In The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21), 202

On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectors

Deep learning based medical image classifiers have shown remarkable prowess in various application areas like ophthalmology, dermatology, pathology, and radiology. However, the acceptance of these Computer-Aided Diagnosis (CAD) systems in real clinical setups is severely limited primarily because their decision-making process remains largely obscure. This work aims at elucidating a deep learning based medical image classifier by verifying that the model learns and utilizes similar disease-related concepts as described and employed by dermatologists. We used a well-trained and high performing neural network developed by REasoning for COmplex Data (RECOD) Lab for classification of three skin tumours, i.e. Melanocytic Naevi, Melanoma and Seborrheic Keratosis and performed a detailed analysis on its latent space. Two well established and publicly available skin disease datasets, PH2 and derm7pt, are used for experimentation. Human understandable concepts are mapped to RECOD image classification model with the help of Concept Activation Vectors (CAVs), introducing a novel training and significance testing paradigm for CAVs. Our results on an independent evaluation set clearly shows that the classifier learns and encodes human understandable concepts in its latent representation. Additionally, TCAV scores (Testing with CAVs) suggest that the neural network indeed makes use of disease-related concepts in the correct way when making predictions. We anticipate that this work can not only increase confidence of medical practitioners on CAD but also serve as a stepping stone for further development of CAV-based neural network interpretation methods.Comment: Accepted for the IEEE International Joint Conference on Neural Networks (IJCNN) 202

Eye-CU: Sleep Pose Classification for Healthcare using Multimodal Multiview Data

Manual analysis of body poses of bed-ridden patients requires staff to continuously track and record patient poses. Two limitations in the dissemination of pose-related therapies are scarce human resources and unreliable automated systems. This work addresses these issues by introducing a new method and a new system for robust automated classification of sleep poses in an Intensive Care Unit (ICU) environment. The new method, coupled-constrained Least-Squares (cc-LS), uses multimodal and multiview (MM) data and finds the set of modality trust values that minimizes the difference between expected and estimated labels. The new system, Eye-CU, is an affordable multi-sensor modular system for unobtrusive data collection and analysis in healthcare. Experimental results indicate that the performance of cc-LS matches the performance of existing methods in ideal scenarios. This method outperforms the latest techniques in challenging scenarios by 13% for those with poor illumination and by 70% for those with both poor illumination and occlusions. Results also show that a reduced Eye-CU configuration can classify poses without pressure information with only a slight drop in its performance.Comment: Ten-page manuscript including references and ten figure