Accurate and interpretable classification of microspectroscopy pixels using artificial neural networks

This paper addresses the problem of classifying materials from microspectroscopy at a pixel level. The challenges lie in identifying discriminatory spectral features and obtaining accurate and interpretable models relating spectra and class labels. We approach the problem by designing a supervised classifier from a tandem of Artificial Neural Network (ANN) models that identify relevant features in raw spectra and achieve high classification accuracy. The tandem of ANN models is meshed with classification rule extraction methods to lower the model complexity and to achieve interpretability of the resulting model. The contribution of the work is in designing each ANN model based on the microspectroscopy hypothesis about a discriminatory feature of a certain target class being composed of a linear combination of spectra. The novelty lies in meshing ANN and decision rule models into a tandem configuration to achieve accurate and interpretable classification results. The proposed method was evaluated using a set of broadband coherent anti-Stokes Raman scattering (BCARS) microscopy cell images (600 000 pixel-level spectra) and a reference four-class rule-based model previously created by biochemical experts. The generated classification rule-based model was on average 85% accurate measured by the DICE pixel label similarity metric, and on average 96% similar to the reference rules measured by the vector cosine metric

Vibrational spectroscopy : are we close to finding a solution for early pancreatic cancer diagnosis?

Pancreatic cancer (PC) is an aggressive and lethal neoplasm, ranking seventh in the world for cancer deaths, with an overall 5-year survival rate of below 10%. The knowledge about PC pathogenesis is rapidly expanding. New aspects of tumor biology, including its molecular and morphological heterogeneity, have been reported to explain the complicated "cross-talk" that occurs between the cancer cells and the tumor stroma or the nature of pancreatic ductal adenocarcinoma-associated neural remodeling. Nevertheless, currently, there are no specific and sensitive diagnosis options for PC. Vibrational spectroscopy (VS) shows a promising role in the development of early diagnosis technology. In this review, we summarize recent reports about improvements in spectroscopic methodologies, briefly explain and highlight the drawbacks of each of them, and discuss available solutions. The important aspects of spectroscopic data evaluation with multivariate analysis and a convolutional neural network methodology are depicted. We conclude by presenting a study design for systemic verification of the VS-based methods in the diagnosis of PC

Deep learning applied to hyperspectral endoscopy for online spectral classification

Abstract: Hyperspectral imaging (HSI) is being explored in endoscopy as a tool to extract biochemical information that may improve contrast for early cancer detection in the gastrointestinal tract. Motion artefacts during medical endoscopy have traditionally limited HSI application, however, recent developments in the field have led to real-time HSI deployments. Unfortunately, traditional HSI analysis methods remain unable to rapidly process the volume of hyperspectral data in order to provide real-time feedback to the operator. Here, a convolutional neural network (CNN) is proposed to enable online classification of data obtained during HSI endoscopy. A five-layered CNN was trained and fine-tuned on a dataset of 300 hyperspectral endoscopy images acquired from a planar Macbeth ColorChecker chart and was able to distinguish between its 18 constituent colors with an average accuracy of 94.3% achieved at 8.8 fps. Performance was then tested on a set of images simulating an endoscopy environment, consisting of color charts warped inside a rigid tube mimicking a lumen. The algorithm proved robust to such variations, with classification accuracies over 90% being obtained despite the variations, with an average drop in accuracy of 2.4% being registered at the points of longest working distance and most inclination. For further validation of the color-based classification system, ex vivo videos of a methylene blue dyed pig esophagus and images of different disease stages in the human esophagus were analyzed, showing spatially distinct color classifications. These results suggest that the CNN has potential to provide color-based classification during real-time HSI in endoscopy

Harnessing Artificial Intelligence for the Next Generation of 3D Printed Medicines

Artificial intelligence (AI) is redefining how we exist in the world. In almost every sector of society, AI is performing tasks with super-human speed and intellect; from the prediction of stock market trends to driverless vehicles, diagnosis of disease, and robotic surgery. Despite this growing success, the pharmaceutical field is yet to truly harness AI. Development and manufacture of medicines remains largely in a ‘one size fits all’ paradigm, in which mass-produced, identical formulations are expected to meet individual patient needs. Recently, 3D printing (3DP) has illuminated a path for on-demand production of fully customisable medicines. Due to its flexibility, pharmaceutical 3DP presents innumerable options during formulation development that generally require expert navigation. Leveraging AI within pharmaceutical 3DP removes the need for human expertise, as optimal process parameters can be accurately predicted by machine learning. AI can also be incorporated into a pharmaceutical 3DP ‘Internet of Things’, moving the personalised production of medicines into an intelligent, streamlined, and autonomous pipeline. Supportive infrastructure, such as The Cloud and blockchain, will also play a vital role. Crucially, these technologies will expedite the use of pharmaceutical 3DP in clinical settings and drive the global movement towards personalised medicine and Industry 4.0

Laser Based Mid-Infrared Spectroscopic Imaging – Exploring a Novel Method for Application in Cancer Diagnosis

A number of biomedical studies have shown that mid-infrared spectroscopic images can provide both morphological and biochemical information that can be used for the diagnosis of cancer. Whilst this technique has shown great potential it has yet to be employed by the medical profession. By replacing the conventional broadband thermal source employed in modern FTIR spectrometers with high-brightness, broadly tuneable laser based sources (QCLs and OPGs) we aim to solve one of the main obstacles to the transfer of this technology to the medical arena; namely poor signal to noise ratios at high spatial resolutions and short image acquisition times. In this thesis we take the first steps towards developing the optimum experimental configuration, the data processing algorithms and the spectroscopic image contrast and enhancement methods needed to utilise these high intensity laser based sources. We show that a QCL system is better suited to providing numerical absorbance values (biochemical information) than an OPG system primarily due to the QCL pulse stability. We also discuss practical protocols for the application of spectroscopic imaging to cancer diagnosis and present our spectroscopic imaging results from our laser based spectroscopic imaging experiments of oesophageal cancer tissue

Deep learning applied to hyperspectral endoscopy for online spectral classification

Abstract: Hyperspectral imaging (HSI) is being explored in endoscopy as a tool to extract biochemical information that may improve contrast for early cancer detection in the gastrointestinal tract. Motion artefacts during medical endoscopy have traditionally limited HSI application, however, recent developments in the field have led to real-time HSI deployments. Unfortunately, traditional HSI analysis methods remain unable to rapidly process the volume of hyperspectral data in order to provide real-time feedback to the operator. Here, a convolutional neural network (CNN) is proposed to enable online classification of data obtained during HSI endoscopy. A five-layered CNN was trained and fine-tuned on a dataset of 300 hyperspectral endoscopy images acquired from a planar Macbeth ColorChecker chart and was able to distinguish between its 18 constituent colors with an average accuracy of 94.3% achieved at 8.8 fps. Performance was then tested on a set of images simulating an endoscopy environment, consisting of color charts warped inside a rigid tube mimicking a lumen. The algorithm proved robust to such variations, with classification accuracies over 90% being obtained despite the variations, with an average drop in accuracy of 2.4% being registered at the points of longest working distance and most inclination. For further validation of the color-based classification system, ex vivo videos of a methylene blue dyed pig esophagus and images of different disease stages in the human esophagus were analyzed, showing spatially distinct color classifications. These results suggest that the CNN has potential to provide color-based classification during real-time HSI in endoscopy

Raman Microspectroscopy and Fluorescence Lifetime Imaging Microscopy based Data-Driven Tissue Discrimination and Diagnostics

The ultimate objective of any new interventional or surgical techniques is to achieve a balance of minimal invasiveness, optimal efficacy, and rapid treatment duration, all while minimizing the risk of complications. A pivotal component in the management of any disease is the distinction between the impacted target structures and the neighboring healthy tissue throughout all medical interventions, encompassing surgical procedures. Such differentiation is paramount in reducing harm to healthy tissue and augmenting the efficacy of the treatment. Within the current therapeutic procedures, innovations in the field of preoperative and postoperative diagnostics contribute to the improved differentiation of benign and malignant tissue structures. On the one hand, sophisticated imaging techniques support an improved surgical decision-making, while on the other hand, histopathological examination methods enable a precise classification of the tissue after surgery. In contrast, intraoperative tissue differentiation has been based on the time-consuming gold standard of frozen section diagnostics for many years. However, by incorporating the supplementary data provided by advanced imaging sensors during surgery, and integrating it with cutting-edge machine learning methodologies, it is feasible to augment the quality of information utilized for tissue differentiation, thereby increasing the precision of the overall process. Another important task of modern medicine is the patient-specific treatment of cancer, as it has been found that different patients do not respond in the same way to drug treatment due to developed resistance mechanisms. This thesis aimed to establish Raman microspectroscopy (RMS) as marker-independent, and non-destructive technique to monitor fibrotic and epigenetic modifications in malign and benign human tissue. Additionally, the potential of RMS and fluorescence lifetime imaging microscopy (FLIM) was evaluated to non-invasively monitor the drug efficacy on patient-derived organoids from cancer patients. Towards this aim, collagen type I (COL I) structures of formalin-fixed paraffin-embedded (FFPE) tissue sections of various fibrotic diseases were compared to respective control tissue sections. Incorporating Raman measurements into the experimental protocol, we conducted routine histological and immunofluorescence (IF) staining techniques to showcase the superior efficacy of RMS when paired with spectral deconvolution. This technique offers a time- and cost-efficient alternative to conventional procedures. For the differentiation of pathological tissue, the identification of epigenetic modes of action is a promising and potentially successful approach. In alignment with IF imaging of the most abundant epigenetic modification of 5-methylcytosine (5mC), Raman spectra of cell nuclei were evaluated using multivariate data analysis. Compared to invasive staining methods, non-invasive RMS showed promising results for the differentiation of pathological tissue changes in cardiac fibrosis and endometriosis. In addition, RMS and FLIM have been used on several bladder and colon cancer organoids to evaluate their potential to monitor patient-specific responses to drug treatment. The results showed both that organoids are generally suitable as a screening platform for drug treatments and that Raman and FLIM have the potential to assess drug sensitivity. This work highlights the potential of RMS for future applications in ex vivo tissue discrimination of fibrotic diseases and identification of epigenetic changes. In addition, this work demonstrates the proof of principle that RMS and FLIM are suitable for monitoring patient-specific responses to medications on organoid models

Near-infrared raman spectroscopy with recursive partitioning techniques for precancer and cancer detection

Master'sMASTER OF ENGINEERIN

From Research to Diagnostic Application of Raman Spectroscopy in Neurosciences: Past and Perspectives

In recent years, Raman spectroscopy has been more and more frequently applied to address research questions in neuroscience. As a non-destructive technique based on inelastic scattering of photons, it can be used for a wide spectrum of applications including neurooncological tumor diagnostics or analysis of misfolded protein aggregates involved in neurodegenerative diseases. Progress in the technical development of this method allows for an increasingly detailed analysis of biological samples and may therefore open new fields of applications. The goal of our review is to provide an introduction into Raman scattering, its practical usage and also commonly associated pitfalls. Furthermore, intraoperative assessment of tumor recurrence using Raman based histology images as well as the search for non-invasive ways of diagnosis in neurodegenerative diseases are discussed. Some of the applications mentioned here may serve as a basis and possibly set the course for a future use of the technique in clinical practice. Covering a broad range of content, this overview can serve not only as a quick and accessible reference tool but also provide more in-depth information on a specific subtopic of interest

Clinical microbiology with multi-view deep probabilistic models

Clinical microbiology is one of the critical topics of this century. Identification and discrimination of microorganisms is considered a global public health threat by the main international health organisations, such as World Health Organisation (WHO) or the European Centre for Disease Prevention and Control (ECDC). Rapid spread, high morbidity and mortality, as well as the economic burden associated with their treatment and control are the main causes of their impact. Discrimination of microorganisms is crucial for clinical applications, for instance, Clostridium difficile (C. diff ) increases the mortality and morbidity of healthcare-related infections. Furthermore, in the past two decades, other bacteria, including Klebsiella pneumoniae (K. pneumonia), have demonstrated a significant propensity to acquire antibiotic resistance mechanisms. Consequently, the use of an ineffective antibiotic may result in mortality. Machine Learning (ML) has the potential to be applied in the clinical microbiology field to automatise current methodologies and provide more efficient guided personalised treatments. However, microbiological data are challenging to exploit owing to the presence of a heterogeneous mix of data types, such as real-valued high-dimensional data, categorical indicators, multilabel epidemiological data, binary targets, or even time-series data representations. This problem, which in the field of ML is known as multi-view or multi-modal representation learning, has been studied in other application fields such as mental health monitoring or haematology. Multi-view learning combines different modalities or views representing the same data to extract richer insights and improve understanding. Each modality or view corresponds to a distinct encoding mechanism for the data, and this dissertation specifically addresses the issue of heterogeneity across multiple views. In the probabilistic ML field, the exploitation of multi-view learning is also known as Bayesian Factor Analysis (FA). Current solutions face limitations when handling high-dimensional data and non-linear associations. Recent research proposes deep probabilistic methods to learn hierarchical representations of the data, which can capture intricate non-linear relationships between features. However, some Deep Learning (DL) techniques rely on complicated representations, which can hinder the interpretation of the outcomes. In addition, some inference methods used in DL approaches can be computationally burdensome, which can hinder their practical application in real-world situations. Therefore, there is a demand for more interpretable, explainable, and computationally efficient techniques for highdimensional data. By combining multiple views representing the same information, such as genomic, proteomic, and epidemiologic data, multi-modal representation learning could provide a better understanding of the microbial world. Hence, in this dissertation, the development of two deep probabilistic models, that can handle current limitations in state-of-the-art of clinical microbiology, are proposed. Moreover, both models are also tested in two real scenarios regarding antibiotic resistance prediction in K. pneumoniae and automatic ribotyping of C. diff in collaboration with the Instituto de Investigación Sanitaria Gregorio Marañón (IISGM) and the Instituto Ramón y Cajal de Investigación Sanitaria (IRyCIS). The first presented algorithm is the Kernelised Sparse Semi-supervised Heterogeneous Interbattery Bayesian Analysis (SSHIBA). This algorithm uses a kernelised formulation to handle non-linear data relationships while providing compact representations through the automatic selection of relevant vectors. Additionally, it uses an Automatic Relevance Determination (ARD) over the kernel to determine the input feature relevance functionality. Then, it is tailored and applied to the microbiological laboratories of the IISGM and IRyCIS to predict antibiotic resistance in K. pneumoniae. To do so, specific kernels that handle Matrix-Assisted Laser Desorption Ionization (MALDI)-Time-Of-Flight (TOF) mass spectrometry of bacteria are used. Moreover, by exploiting the multi-modal learning between the spectra and epidemiological information, it outperforms other state-of-the-art algorithms. Presented results demonstrate the importance of heterogeneous models that can analyse epidemiological information and can automatically be adjusted for different data distributions. The implementation of this method in microbiological laboratories could significantly reduce the time required to obtain resistance results in 24-72 hours and, moreover, improve patient outcomes. The second algorithm is a hierarchical Variational AutoEncoder (VAE) for heterogeneous data using an explainable FA latent space, called FA-VAE. The FA-VAE model is built on the foundation of the successful KSSHIBA approach for dealing with semi-supervised heterogeneous multi-view problems. This approach further expands the range of data domains it can handle. With the ability to work with a wide range of data types, including multilabel, continuous, binary, categorical, and even image data, the FA-VAE model offers a versatile and powerful solution for real-world data sets, depending on the VAE architecture. Additionally, this model is adapted and used in the microbiological laboratory of IISGM, resulting in an innovative technique for automatic ribotyping of C. diff, using MALDI-TOF data. To the best of our knowledge, this is the first demonstration of using any kind of ML for C. diff ribotyping. Experiments have been conducted on strains of Hospital General Universitario Gregorio Marañón (HGUGM) to evaluate the viability of the proposed approach. The results have demonstrated high accuracy rates where KSSHIBA even achieved perfect accuracy in the first data collection. These models have also been tested in a real-life outbreak scenario at the HGUGM, where successful classification of all outbreak samples has been achieved by FAVAE. The presented results have not only shown high accuracy in predicting each strain’s ribotype but also revealed an explainable latent space. Furthermore, traditional ribotyping methods, which rely on PCR, required 7 days while FA-VAE has predicted equal results on the same day. This improvement has significantly reduced the time response by helping in the decision-making of isolating patients with hyper-virulent ribotypes of C. diff on the same day of infection. The promising results, obtained in a real outbreak, have provided a solid foundation for further advancements in the field. This study has been a crucial stepping stone towards realising the full potential of MALDI-TOF for bacterial ribotyping and advancing our ability to tackle bacterial outbreaks. In conclusion, this doctoral thesis has significantly contributed to the field of Bayesian FA by addressing its drawbacks in handling various data types through the creation of novel models, namely KSSHIBA and FA-VAE. Additionally, a comprehensive analysis of the limitations of automating laboratory procedures in the microbiology field has been carried out. The shown effectiveness of the newly developed models has been demonstrated through their successful implementation in critical problems, such as predicting antibiotic resistance and automating ribotyping. As a result, KSSHIBA and FA-VAE, both in terms of their technical and practical contributions, signify noteworthy progress both in the clinical and the Bayesian statistics fields. This dissertation opens up possibilities for future advancements in automating microbiological laboratories.La microbiología clínica es uno de los temas críticos de este siglo. La identificación y discriminación de microorganismos se considera una amenaza mundial para la salud pública por parte de las principales organizaciones internacionales de salud, como la Organización Mundial de la Salud (OMS) o el Centro Europeo para la Prevención y Control de Enfermedades (ECDC). La rápida propagación, alta morbilidad y mortalidad, así como la carga económica asociada con su tratamiento y control, son las principales causas de su impacto. La discriminación de microorganismos es crucial para aplicaciones clínicas, como el caso de Clostridium difficile (C. diff ), el cual aumenta la mortalidad y morbilidad de las infecciones relacionadas con la atención médica. Además, en las últimas dos décadas, otros tipos de bacterias, incluyendo Klebsiella pneumoniae (K. pneumonia), han demostrado una propensión significativa a adquirir mecanismos de resistencia a los antibióticos. En consecuencia, el uso de un antibiótico ineficaz puede resultar en un aumento de la mortalidad. El aprendizaje automático (ML) tiene el potencial de ser aplicado en el campo de la microbiología clínica para automatizar las metodologías actuales y proporcionar tratamientos personalizados más eficientes y guiados. Sin embargo, los datos microbiológicos son difíciles de explotar debido a la presencia de una mezcla heterogénea de tipos de datos, tales como datos reales de alta dimensionalidad, indicadores categóricos, datos epidemiológicos multietiqueta, objetivos binarios o incluso series temporales. Este problema, conocido en el campo del aprendizaje automático (ML) como aprendizaje multimodal o multivista, ha sido estudiado en otras áreas de aplicación, como en el monitoreo de la salud mental o la hematología. El aprendizaje multivista combina diferentes modalidades o vistas que representan los mismos datos para extraer conocimientos más ricos y mejorar la comprensión. Cada vista corresponde a un mecanismo de codificación distinto para los datos, y esta tesis aborda particularmente el problema de la heterogeneidad multivista. En el campo del aprendizaje automático probabilístico, la explotación del aprendizaje multivista también se conoce como Análisis de Factores (FA) Bayesianos. Las soluciones actuales enfrentan limitaciones al manejar datos de alta dimensionalidad y correlaciones no lineales. Investigaciones recientes proponen métodos probabilísticos profundos para aprender representaciones jerárquicas de los datos, que pueden capturar relaciones no lineales intrincadas entre características. Sin embargo, algunas técnicas de aprendizaje profundo (DL) se basan en representaciones complejas, dificultando así la interpretación de los resultados. Además, algunos métodos de inferencia utilizados en DL pueden ser computacionalmente costosos, obstaculizando su aplicación práctica. Por lo tanto, existe una demanda de técnicas más interpretables, explicables y computacionalmente eficientes para datos de alta dimensionalidad. Al combinar múltiples vistas que representan la misma información, como datos genómicos, proteómicos y epidemiológicos, el aprendizaje multimodal podría proporcionar una mejor comprensión del mundo microbiano. Dicho lo cual, en esta tesis se proponen el desarrollo de dos modelos probabilísticos profundos que pueden manejar las limitaciones actuales en el estado del arte de la microbiología clínica. Además, ambos modelos también se someten a prueba en dos escenarios reales relacionados con la predicción de resistencia a los antibióticos en K. pneumoniae y el ribotipado automático de C. diff en colaboración con el IISGM y el IRyCIS. El primer algoritmo presentado es Kernelised Sparse Semi-supervised Heterogeneous Interbattery Bayesian Analysis (SSHIBA). Este algoritmo utiliza una formulación kernelizada para manejar correlaciones no lineales proporcionando representaciones compactas a través de la selección automática de vectores relevantes. Además, utiliza un Automatic Relevance Determination (ARD) sobre el kernel para determinar la relevancia de las características de entrada. Luego, se adapta y aplica a los laboratorios microbiológicos del IISGM y IRyCIS para predecir la resistencia a antibióticos en K. pneumoniae. Para ello, se utilizan kernels específicos que manejan la espectrometría de masas Matrix-Assisted Laser Desorption Ionization (MALDI)-Time-Of-Flight (TOF) de bacterias. Además, al aprovechar el aprendizaje multimodal entre los espectros y la información epidemiológica, supera a otros algoritmos de última generación. Los resultados presentados demuestran la importancia de los modelos heterogéneos ya que pueden analizar la información epidemiológica y ajustarse automáticamente para diferentes distribuciones de datos. La implementación de este método en laboratorios microbiológicos podría reducir significativamente el tiempo requerido para obtener resultados de resistencia en 24-72 horas y, además, mejorar los resultados para los pacientes. El segundo algoritmo es un modelo jerárquico de Variational AutoEncoder (VAE) para datos heterogéneos que utiliza un espacio latente con un FA explicativo, llamado FA-VAE. El modelo FA-VAE se construye sobre la base del enfoque de KSSHIBA para tratar problemas semi-supervisados multivista. Esta propuesta amplía aún más el rango de dominios que puede manejar incluyendo multietiqueta, continuos, binarios, categóricos e incluso imágenes. De esta forma, el modelo FA-VAE ofrece una solución versátil y potente para conjuntos de datos realistas, dependiendo de la arquitectura del VAE. Además, este modelo es adaptado y utilizado en el laboratorio microbiológico del IISGM, lo que resulta en una técnica innovadora para el ribotipado automático de C. diff utilizando datos MALDI-TOF. Hasta donde sabemos, esta es la primera demostración del uso de cualquier tipo de ML para el ribotipado de C. diff. Se han realizado experimentos en cepas del Hospital General Universitario Gregorio Marañón (HGUGM) para evaluar la viabilidad de la técnica propuesta. Los resultados han demostrado altas tasas de precisión donde KSSHIBA incluso logró una clasificación perfecta en la primera colección de datos. Estos modelos también se han probado en un brote real en el HGUGM, donde FA-VAE logró clasificar con éxito todas las muestras del mismo. Los resultados presentados no solo han demostrado una alta precisión en la predicción del ribotipo de cada cepa, sino que también han revelado un espacio latente explicativo. Además, los métodos tradicionales de ribotipado, que dependen de PCR, requieren 7 días para obtener resultados mientras que FA-VAE ha predicho resultados correctos el mismo día del brote. Esta mejora ha reducido significativamente el tiempo de respuesta ayudando así en la toma de decisiones para aislar a los pacientes con ribotipos hipervirulentos de C. diff el mismo día de la infección. Los resultados prometedores, obtenidos en un brote real, han sentado las bases para nuevos avances en el campo. Este estudio ha sido un paso crucial hacia el despliegue del pleno potencial de MALDI-TOF para el ribotipado bacteriana avanzado así nuestra capacidad para abordar brotes bacterianos. En conclusión, esta tesis doctoral ha contribuido significativamente al campo del FA Bayesiano al abordar sus limitaciones en el manejo de tipos de datos heterogéneos a través de la creación de modelos noveles, concretamente, KSSHIBA y FA-VAE. Además, se ha llevado a cabo un análisis exhaustivo de las limitaciones de la automatización de procedimientos de laboratorio en el campo de la microbiología. La efectividad de los nuevos modelos, en este campo, se ha demostrado a través de su implementación exitosa en problemas críticos, como la predicción de resistencia a los antibióticos y la automatización del ribotipado. Como resultado, KSSHIBA y FAVAE, tanto en términos de sus contribuciones técnicas como prácticas, representan un progreso notable tanto en los campos clínicos como en la estadística Bayesiana. Esta disertación abre posibilidades para futuros avances en la automatización de laboratorios microbiológicos.Programa de Doctorado en Multimedia y Comunicaciones por la Universidad Carlos III de Madrid y la Universidad Rey Juan CarlosPresidente: Juan José Murillo Fuentes.- Secretario: Jerónimo Arenas García.- Vocal: María de las Mercedes Marín Arriaz