Deep learning applied to hyperspectral endoscopy for online spectral classification

Abstract: Hyperspectral imaging (HSI) is being explored in endoscopy as a tool to extract biochemical information that may improve contrast for early cancer detection in the gastrointestinal tract. Motion artefacts during medical endoscopy have traditionally limited HSI application, however, recent developments in the field have led to real-time HSI deployments. Unfortunately, traditional HSI analysis methods remain unable to rapidly process the volume of hyperspectral data in order to provide real-time feedback to the operator. Here, a convolutional neural network (CNN) is proposed to enable online classification of data obtained during HSI endoscopy. A five-layered CNN was trained and fine-tuned on a dataset of 300 hyperspectral endoscopy images acquired from a planar Macbeth ColorChecker chart and was able to distinguish between its 18 constituent colors with an average accuracy of 94.3% achieved at 8.8 fps. Performance was then tested on a set of images simulating an endoscopy environment, consisting of color charts warped inside a rigid tube mimicking a lumen. The algorithm proved robust to such variations, with classification accuracies over 90% being obtained despite the variations, with an average drop in accuracy of 2.4% being registered at the points of longest working distance and most inclination. For further validation of the color-based classification system, ex vivo videos of a methylene blue dyed pig esophagus and images of different disease stages in the human esophagus were analyzed, showing spatially distinct color classifications. These results suggest that the CNN has potential to provide color-based classification during real-time HSI in endoscopy

Deep learning applied to hyperspectral endoscopy for online spectral classification

Abstract: Hyperspectral imaging (HSI) is being explored in endoscopy as a tool to extract biochemical information that may improve contrast for early cancer detection in the gastrointestinal tract. Motion artefacts during medical endoscopy have traditionally limited HSI application, however, recent developments in the field have led to real-time HSI deployments. Unfortunately, traditional HSI analysis methods remain unable to rapidly process the volume of hyperspectral data in order to provide real-time feedback to the operator. Here, a convolutional neural network (CNN) is proposed to enable online classification of data obtained during HSI endoscopy. A five-layered CNN was trained and fine-tuned on a dataset of 300 hyperspectral endoscopy images acquired from a planar Macbeth ColorChecker chart and was able to distinguish between its 18 constituent colors with an average accuracy of 94.3% achieved at 8.8 fps. Performance was then tested on a set of images simulating an endoscopy environment, consisting of color charts warped inside a rigid tube mimicking a lumen. The algorithm proved robust to such variations, with classification accuracies over 90% being obtained despite the variations, with an average drop in accuracy of 2.4% being registered at the points of longest working distance and most inclination. For further validation of the color-based classification system, ex vivo videos of a methylene blue dyed pig esophagus and images of different disease stages in the human esophagus were analyzed, showing spatially distinct color classifications. These results suggest that the CNN has potential to provide color-based classification during real-time HSI in endoscopy

Information Extraction Techniques in Hyperspectral Imaging Biomedical Applications

Hyperspectral imaging (HSI) is a technology able to measure information about the spectral reflectance or transmission of light from the surface. The spectral data, usually within the ultraviolet and infrared regions of the electromagnetic spectrum, provide information about the interaction between light and different materials within the image. This fact enables the identification of different materials based on such spectral information. In recent years, this technology is being actively explored for clinical applications. One of the most relevant challenges in medical HSI is the information extraction, where image processing methods are used to extract useful information for disease detection and diagnosis. In this chapter, we provide an overview of the information extraction techniques for HSI. First, we introduce the background of HSI, and the main motivations of its usage for medical applications. Second, we present information extraction techniques based on both light propagation models within tissue and machine learning approaches. Then, we survey the usage of such information extraction techniques in HSI biomedical research applications. Finally, we discuss the main advantages and disadvantages of the most commonly used image processing approaches and the current challenges in HSI information extraction techniques in clinical applications

A robust dynamic classifier selection approach for hyperspectral images with imprecise label information

Supervised hyperspectral image (HSI) classification relies on accurate label information. However, it is not always possible to collect perfectly accurate labels for training samples. This motivates the development of classifiers that are sufficiently robust to some reasonable amounts of errors in data labels. Despite the growing importance of this aspect, it has not been sufficiently studied in the literature yet. In this paper, we analyze the effect of erroneous sample labels on probability distributions of the principal components of HSIs, and provide in this way a statistical analysis of the resulting uncertainty in classifiers. Building on the theory of imprecise probabilities, we develop a novel robust dynamic classifier selection (R-DCS) model for data classification with erroneous labels. Particularly, spectral and spatial features are extracted from HSIs to construct two individual classifiers for the dynamic selection, respectively. The proposed R-DCS model is based on the robustness of the classifiers’ predictions: the extent to which a classifier can be altered without changing its prediction. We provide three possible selection strategies for the proposed model with different computational complexities and apply them on three benchmark data sets. Experimental results demonstrate that the proposed model outperforms the individual classifiers it selects from and is more robust to errors in labels compared to widely adopted approaches

Artificial Intelligence in Medicine and Healthcare: applications, availability and societal impact

Comisión Europea. Joint Research Centre. Serie: JRC Science for Police ReportThis report reviews and classifies the current and near-future applications of Artificial Intelligence (AI) in Medicine and Healthcare according to their ethical and societal impact and the availability level of the various technological implementations. It provides conceptual foundations for well-informed policy-oriented work, research, and forward-looking activities that address the opportunities and challenges created in the field of AI in Medicine and Healthcare. This report is aimed for policy developers, but it also makes contributions that are of interest for researchers studying the impact and the future of AI on Healthcare, for scientific and technological stakeholders in this field and for the general public.This report is based on an analysis of the state of the art of research and technology, including software, personal monitoring devices, genetic tests and editing tools, personalized digital models, online platforms, augmented reality devices, and surgical and companion robotics. From this analysis, it is presented the concept of “extended personalized medicine”, and it is explored the public perception of medical AI systems, and how they show, simultaneously, extraordinary opportunities and drawbacks. In addition, this report addresses the transformation of the roles of doctors and patients in an age of ubiquitous information and identifies three main paradigms in AI-supported Medicine: “fake-based”, “patient-generated”, and “scientifically tailored” views.This Report presents:- An updated overview of the many aspects related to the social impact of Artificial Intelligence and its applications in Medicine and Health. A new ‘Technology Availability Scale’ is defined to evaluate and compare their current status.- Recent examples of the growing social concerns and debates in the general press, social media and other web-bases sources.- A ‘Visual Overview of AI and AI-mediated technologies in Medicine and Healthcare’, in which two figures show, respeComisión Europea. Joint Research Centr

Artificial Intelligence in Medicine and Healthcare: applications, availability and societal impact

This report reviews and classifies the current and near-future applications of Artificial Intelligence (AI) in Medicine and Healthcare according to their ethical and societal impact and the availability level of the various technological implementations. It provides conceptual foundations for well-informed policy-oriented work, research, and forward-looking activities that address the opportunities and challenges created in the field of AI in Medicine and Healthcare. This report is aimed for policy developers, but it also makes contributions that are of interest for researchers studying the impact and the future of AI on Healthcare, for scientific and technological stakeholders in this field and for the general public. This report is based on an analysis of the state of the art of research and technology, including software, personal monitoring devices, genetic tests and editing tools, personalized digital models, online platforms, augmented reality devices, and surgical and companion robotics. From this analysis, it is presented the concept of “extended personalized medicine”, and it is explored the public perception of medical AI systems, and how they show, simultaneously, extraordinary opportunities and drawbacks. In addition, this report addresses the transformation of the roles of doctors and patients in an age of ubiquitous information and identifies three main paradigms in AI-supported Medicine: “fake-based”, “patient-generated”, and “scientifically tailored” views. This Report presents: - An updated overview of the many aspects related to the social impact of Artificial Intelligence and its applications in Medicine and Health. A new ‘Technology Availability Scale’ is defined to evaluate and compare their current status. - Recent examples of the growing social concerns and debates in the general press, social media and other web-bases sources. - A ‘Visual Overview of AI and AI-mediated technologies in Medicine and Healthcare’, in which two figures show, respectively, a (newly proposed) classification according to their ethical and social impact, and the most relevant ethical and social aspects considered for such classification. Some key questions, controversies, significant, and conflicting issues are outlined for each aspect. - A ‘Structured Overview’, with a sorted list of technologies and their implementations, including perspectives, conflicting views and potential pitfalls, and a corresponding, extensive list of references. - A conclusive set of policy challenges, namely the need of informed citizens, key aspects (of AI and AI-mediated technologies in Medicine and Healthcare) to evaluate, and some recommendations towards a European leadership in this sector. - We finally relate our study with an update on the use of AI technologies to fight the SARS-CoV-2 virus and COVID-19 pandemic disease.JRC.A.5-Scientific Developmen

Development of deep learning methods for head and neck cancer detection in hyperspectral imaging and digital pathology for surgical guidance

Surgeons performing routine cancer resections utilize palpation and visual inspection, along with time-consuming microscopic tissue analysis, to ensure removal of cancer. Despite this, inadequate surgical cancer margins are reported for up to 10-20% of head and neck squamous cell carcinoma (SCC) operations. There exists a need for surgical guidance with optical imaging to ensure complete cancer resection in the operating room. The objective of this dissertation is to evaluate hyperspectral imaging (HSI) as a non-contact, label-free optical imaging modality to provide intraoperative diagnostic information. For comparison of different optical methods, autofluorescence, RGB composite images synthesized from HSI, and two fluorescent dyes are also acquired and investigated for head and neck cancer detection. A novel and comprehensive dataset was obtained of 585 excised tissue specimens from 204 patients undergoing routine head and neck cancer surgeries. The first aim was to use SCC tissue specimens to determine the potential of HSI for surgical guidance in the challenging task of head and neck SCC detection. It is hypothesized that HSI could reduce time and provide quantitative cancer predictions. State-of-the-art deep learning algorithms were developed for SCC detection in 102 patients and compared to other optical methods. HSI detected SCC with a median AUC score of 85%, and several anatomical locations demonstrated good SCC detection, such as the larynx, oropharynx, hypopharynx, and nasal cavity. To understand the ability of HSI for SCC detection, the most important spectral features were calculated and correlated with known cancer physiology signals, notably oxygenated and deoxygenated hemoglobin. The second aim was to evaluate HSI for tumor detection in thyroid and salivary glands, and RGB images were synthesized using the spectral response curves of the human eye for comparison. Using deep learning, HSI detected thyroid tumors with 86% average AUC score, which outperformed fluorescent dyes and autofluorescence, but HSI-synthesized RGB imagery performed with 90% AUC score. The last aim was to develop deep learning algorithms for head and neck cancer detection in hundreds of digitized histology slides. Slides containing SCC or thyroid carcinoma can be distinguished from normal slides with 94% and 99% AUC scores, respectively, and SCC and thyroid carcinoma can be localized within whole-slide images with 92% and 95% AUC scores, respectively. In conclusion, the outcomes of this thesis work demonstrate that HSI and deep learning methods could aid surgeons and pathologists in detecting head and neck cancers.Ph.D